[douglas]

The troop transport's design must have its strategy set to Capture Planet. You can change design strategies in the Ship Design screen (F3).

Edit: Also, if the enemy planet has weapon platforms on it, it is possible for your ships to glass the planet while trying to take out the weapon platforms. Ships will not fire on defenseless planets if a troop-carrying ship with the right strategy is present, but that check is only performed between entire volleys (not turns, just one ship). If the first shot in a volley finishes the last weapon platform, the remaining shots your ship has might be enough to destroy the colony. This is especially a problem with using missile weapons, because your ships won't stop launching until the last weapon platform is gone, so you'll almost certainly have a lot of missiles still in flight.