Each province has a type of inhabitants, and when you conquer it you can recruit the armies that go with it : lizard warriors, amazons, hoburg militias
, and so on. Even when you capture an enemy's capital, you can recruit the armies that go with the province type. But not the special armies.
Some of it is common sense : there is no way Ermor would ever be allowed to recruit Marignon's paladins, right ? I mean, if they tried, they would at most get ordinary knights (which they can get in the right province)
But the real reason is probably game balance : some races get good mages, other get good infantries, if you start getting both you are getting close to invincible ...