Fatigue is not very realistic!?

[Kobal2]

It's an abstraction.
A giant swinging a 60 pound sword and a hobbit wielding a toothpick don't expend the same amount of effort and exert the same amount of force. But from each of their reference points, they do exert the same amount of fatigue.

Don't consider fatigue as a universal score, but a relative one. One point of giant fatigue is WAAAAAY more tiring than one point of human fatigue. But they're giants with WAAAAY more energy and muscle power, so it's a wash.