hey this is my pet problem!
some suggestions:
if the planet has few populations, then there is a good chance that there is not much defenses. You can have a special troop transport (smartly labelled Assault Transport) with 2-3 point defenses and a lot of shieds. Normally 2 or 3 of these will be enough to land on the planets, even if there is defense firing on them (well you can loose all but one, but that enough). The interest of PD weapons is that they should kill seekers and fighters without harming the planet (in case there is not too much enemy).
For this strategy to works, you should not have attack ships in the fleet, only transports attacking the planet. You must then be sure to have no enemy ships moving in the planet sector during the turn (or phase if you are precise to synchronize).
If the planet has many population, your ships should be able to fire there Last volley without killing all the pop, so you dont have to use the 1st strategy.
Lastly, if you play solo, then you can increase by a tenfold the resistant of population against bombing (in settings.txt iirc). This will help you in not glassing the planet with the Last volley of your ships.
Generally speaking, if you have a ship with troops on capture planet, then you should now be able to take one, with the only problem remaining being the Last volley killing all the pop.
Re: How to NOT glass a planet
Hybrid Mode
