I think only the Resupply minister needs to be turned off. I don't know for sure since I never used Ministers since v.0.20...
But Resupply minister is the only one I know of that does any resupplying. The Fleet minister only controls the creation/disbanding of fleets, but this would be the obvious choice for those not familiar with the workings of the game. Also check that the fleet doesn't have an individual minister (it will have a little head icon if it does), that might cause problems.