Flying already has a huge numbers of advantage over walking troops (the primary ones being able to strike past domains not through them, like even hightly mobile land armies). Being able to circumvent the Siege phase of a castle would be too much.

All flying troops would have to be dramatically rebalanced to account for this, making them overall much weaker.

Even though it's illogical, I prefer it this way so that having a castle is an effective speedbump for invasions.