No, they didn't grew.
When you colonize a planet, the atmosphere determines if you will be able to use it to its full capacity or just 1/5 of it. When you start your race has only one breathable atmosphere (e.g. Oxygen). So you should look to colonize oxygen planets to use its max potential.
You can colonize other planets with different atmospheres, but they will be 'domed' (people will not live freely of the surface, but in small domed cities), and they will have only use 1/5 of planets full capacity.
When you conquer (not destroy) colonies of other empires, you will also conquer their population. Now, if that population breathes different atmosphere (e.g. Hydrogen) you can use that population to colonize worlds with Hydrogen atmosphere. After that these Hydrogen worlds will use its full capacity with conquered population just like Oxygen worlds use theirs with your population. This way you can increase the capacity of your colonies.
Of course, the maximum capacity of planet is also determined by its size. That can't be increased.