When the attacking force leaves the fort (without you taking the castle back by force) it's still reported as sieged even if there is no one there. It's easy to reproduce. It works as if the attacker was sieging with 0 units until the turn is over.
You also get a message from the attacking side that you don't breach the wall anymore.
I don't know if that's intended or not, though.
