[Alneyan]

The best I seem able to do is to force them to capture the planet, even if they have weapons (at least, I think it worked this way). Otherwise, the troop transport will just fire at the planet until it either runs out of ammo, or the planet has been fully glassed.

It will not be of much help to you, but... Capturing planets work fine when there is no treaty, and so the problem is likely of having both weapons (a Warship role) and troops on board the ship. You should be able to force the ship to act as a warship, or as a troop transport, but both during the same battle doesn't seem possible (unless I am missing something here).