[Alneyan]

If the siege is maintained, the castle walls are not repaired. Leaving a single commander besieging the fort would be enough to make sure the walls are not repaired. In one game, independents besieged one of my castles, and brought it down to 0 strength. When the independents were dispatched by another player, the walls were still down to 0 strength, before the "repair/destruction of the walls" phase.

However, I do not know what happens if you take all your army with you. My guess is that your reinforcing army would move in before the siege can be broken by the besiegers, as movement between friendly provinces should happen first. But don't quote me on that.