Agreed, if any new buildings are ever introduced keep them minimal.
irregardless of that I'm still going to suggest one
Have a setting in game creation where no castles can be built beyond the capital. (or 1,2,3, etc beyond capital). At the same time have a "recruitment/training center" or somesuch where troops can be recruited.
Initially, sieging 50 AI castles in SP is a pain. I think it would also change MP by speeding it up a bit more and reducing castle grind on nations like LA Ermor. Give the benefits of a castle if it works (+resource %, +income %, +supply %, etc) there would just not be a siege phase. I'd probably still keep the same cost to limit them, or maybe 20% less...but it would speed up games a bit. Possibly upgrade castles at the same time as they're more valuable now..shrug.
If the above was ever implemented, possibly add a garrison command to an army. Garrison would just reduce income/supply needed to pay for turn by turn upkeep of the units. They would almost turn into PD, just not regenerating.