Ignoring for the moment that real castles took years or decades or lifetimes to build... yes it's true a castle would often take a lot of damage in the process of being taken by force. On the other hand, both the defenders and the attackers would likely have prepared a lot of extra defenses - new interior and exterior walls, seige engines, etc., and after the defense fell, all that survived would be in the hands of the victorious attackers. A prudent commander expecting a counter-attack could re-arrange it all for a new defense, and perhaps have it as strong, or stronger, a position as it was originally. Just a thought.
All in all, it seems like a lot of speculation about an abstract system. Perhaps the ideal way to handle it would be a random table of possible outcomes of a fort capture, such as:
25% Fort is destroyed by the capture.
10% Fort is crippled and acts at 25% in all ways, until a commander takes time and spends gold (75% original time/cost) to repair the fort.
10% Fort is badly damaged - as above but 50% and cheaper (50%) to fix.
10% Fort is damaged - as above but 75% and 25% to fix.
10% Fort defends at full strength but has no administrative or supply effects until 50% of original fort cost and one turn is spent.
25% Fort is essentially captured intact.
10% Fort is now a stronger defensive position than it was before the seige, due to added defensive works, equipment, and practical experience retained by the new castellan - add 25% to defense value.
Or, just call it a wash and leave it as is.
PvK