Impressions can be misleading. Keeping specific counts tends toward the AI going off script for good reasons more than not. Of course, anytime we disagree with it that tends to stick in our memory for a long time.
However, you will find that many of the advanced players will take scouts along with any walking army that has mages. The scouts can stay hidden out of combat and hold things. Gems, blood slaves, equipment for breaking castles, etc etc. Its abit more micromangment but it is a way to handle the situation. Instead of having mages haul enough gems for a long war, only give them enough for the next fight. Even better if you have stealth flyers who can quickly replenish