No, they also can use gems just to cut fatigue costs. Often seems pretty wasteful, especially if you were hoping the gem stash would Last multiple battles. So sometimes it is a good idea to have non-casters (hiding scouts are a good choice) carry a gem supply, and only hand out gems that you don't mind them using in the immediate battle.
If you give a caster gems, on most spells cast, he will use gems up to his caster level to reduce fatigue. Since with Phoenix Power, your mage is an F3 mage, he can use 3 gems per cast, so will cast the spell using 3 gems nearly every time.