[Cohen]

Building a lot of Castles is common in middle ages, every "province" had his castle, big or small ...

Building a lot of the requires a lot of turns, gold and efforts, not to count that a successfull raid burn out an half built castle.

However probably Castles could be reviewed.
Effectively Watchtower and Mausoleum aren't real castle, and didn't provided any real defence to the province.
A wizard tower too don't provide a good defence.
My proposal could get some code work ... but if we count Watchtower as a patrol bonus unit instead of a castle (probably none will choose it anymore however) for defence issue (so in you conquer the province you take immediately the watchtower and you don't need to siege and storm), but you can recruit as a castle.
Mausoleum should work the same for defence and build, but should give a bonus when casting Death spells ... a small bonus like a 10% gem discount.
Wizard Tower should give a small bonus when casting a remote or summoning spell ... perhaps a 10% discount on gems, rounded down, because the tower is suited to support magic channeling (ie a Seeking Arrow will still cost 3 gems, a Fire from Afar will drop from 10 to 9 gems, a Summon Lamia Queen from 15 to 14 because it's rounded down the discount). Wizard Tower doesn't count as castle for defence meaning.

I'd add some Administration to all other castles.