Yep, I agree as well. I do not mind castling. In fact, since I was raised in the german country side I am pretty used to see a castle around every other corner. Nevertheless, I think that some castle types need to be made much more attractive than they are right now, especially the 4-5 turners!
BUT:
I'd also say that it is ridiculous that a single limping milita man suffices to defend a castle against immediate storming! I'd say that if the attacking force not only breaks the gates immediately on the first turn, but still has the same amount of siege points additionally leftover, it should be allowed to storm immediately. So those watchtowers will still protect a temple against GhostRiders as it currently does, but it wouldnt be such a hindrance to a proper attacking force as it is now (because of long sieges, ghost riders and fires from afar,...)! And nothing would be different for a moderately manned castle having a sensible amount of defense as well.
I would say that anything that encourages attacking with masses of national troops is a good thing. And if it encourages to choose sturdy castles as well, well... 