Your dominion will tear down your homeprovince so quick it will be hard to recruit many later in the game (when you need them the most). You would have to rely on taking provinces that can stand your destructive dominion, spend the gold and turns to create a castle then recruit them. While not inconcievable, harpies die quite quickly and you'll need a good clip of them to create the effect you need (attacking the back).

What I'm saying is don't rely on anything you can build, think of it as 'bonus' and not as 'standard'.