I know provinces are big, but ancient/medieval armies are very big, too. Especially considering size of supply lines. Sitting next to a castle shouldn't be healthy. Castle is good base of operations. If you just ignore it, it can launch an attack on your flank or supply lines.
http://en.wikipedia.org/wiki/Medieva...r_and_foraging
http://en.wikipedia.org/wiki/Medieva...#Supply_trains
I understand it depends on how much abstraction you're going to accept. But current model (army on province = siege) sort of contradicts the 'provinces are big' argument.