Neutral empire would definetly be limiting. It's not real fun though.
Plus you are really screwed if you get stuck in a system with only a planet or two.
I could see them being somewhat viable if you allow them to send ships out of their homesystem, only not colonize. And if they find a really nice system with lot's of planets they can abandon their original home system and relocate.