I'm guessing that your ally launched them in combat, "defending" your planet
.
You could build a minesweeper, and leave it in orbit just incase. You may also be able to mothball it, then unmothball and sweep as soon as you get word that you're not allies anymore. As long as no ships enter the sector, your ships within the sector may be safe.