[Sombre]

Or they could have halberdiers, fliers etc in the castle.

Rather than just throwing men at it, I'd go for some specialised strategy such as a magical item etc. Or you could withdraw and let the enemy come out, then resiege a few turns later.