It seems to me that when empires get really big, and super powerful, they tend to crumble from within. The citizens get used to being top of the heap, and just start to assume they will be forever (Assyrians, Babalonians, Greeks, Romans, Ottomans, British...). The result is a growing decadence within the citizenry, and the whole system becomes less productive and far more wasteful.
i.e.
As a part of the 5th X, I'd love to see a realistic lazy-factor for the most powerful empire. When my empire hits a score of say 5x more than the 2nd place empire, I'd get a message that the citizenry would rather play game boy than work (productivity and build rate drops 5%, maint. increases 50%). When I hit say 10x more than the 2nd place empire, everyone would rather go to holo-vid theatres than work (prod./build rates drop 10%, maint. increases 100%)plus the citizens on my home world would rather not pay taxes anymore so they rebel, etc.
And _then_ the barbarian hordes/space pirates hit! Oh sure, I've got the ships to kick his one-eyed, peg-legged, parrot lovin' butt across the galaxy... except every time I pull ships away from my planets they revolt! Now it's a challenge
