Well - dunno if this was the case, but if you have no troops inside the castle guarding it, invaders can move in and take it in the same turn they move into the province. After all, no one's inside.
__________________
Wormwood and wine, and the bitter taste of ashes.