[Norfleet]

Well, presumably, forts are only difficult to build and repair when enemies are actually shooting at you. If nobody's around to stop you, a fort can be built from scratch in a single turn by a lone scout, apparently assisted by contracted peasants, the undead, etc. Since the attacker only destroys gates of forts, rather than levelling the entire place to the ground in the process of attacking it, the timeframe involved for such a repair certainly cannot be greater than constructing an entirely new fort to begin with!

Thus, the present system works, and seems to be applied in most games that have sieging: The effects of sieging tend to disappear quickly after the siege is lifted.