For more "realism" there's always
Proportions. Colony-building is a *lot* slower, and taking an enemy homeworld is a cast-iron bear.
Of course, for the ultimate in difficulty, there's nothing like going over to SEIV-PBW and matching wits with real sentient opponents. (Sadly, something I no longer have the time to do...)
(Edit - added link)