There are a couple of issues.
The AI tends to invade the planet the fleet encounters in the combat window first. Now, you may have ordered it to attack the bigger one, and some people insist that
never fails -- but people do report this sort of failure often enough to make me think it may be a mild bug.
Once you've conquered a planet, weapon platforms on the opposing planets now may target each other. If the fleet formation prevents the troop ship from getting into position quickly enough, well, fortunes of war and all that, chaps. Cheers.
I've been playing FQM-Deluxe with the TDM modpack. No deliberate glassing, must try to invade at least. With all those breatheable moons -- it's tough.