I wouldn't worry too much. At some point everyone will look at the server list and realise everyone is waiting on them.
The important thing to remember is not to make them wait twice in a row.
I am playing a different game at the moment were some guy is about to get triple teamed because he is always the last guy to submit his turn. In fact he would get quad teamed except the 4th guy can't get his armies in to fight.