Getting stealthy priests to preach stealthily in your own province?

[tinkthank]

I enjoy stealth. I also enjoy hit-and-run tactics. I just love stealthy priests.

With some nations who can do this well (Pan, Marignon, etc. etc.), I have a stealthy priest in province X preaching behind enemy lines, my favorite.

When I think the time is ripe, my priest tells me when the province looks nicely undefended, I will send/sneak in some fliers/teleporters/magic attacks/stealthy troops and take the province.

Once the province is mine, I like to have my troops leave and sneak/fly/etc. somewhere else; I like to put 1 point of PD up (just so I can make sure to watch the battle and see who came to reclaim in case my priest also leaves) and go wreak havoc somewhere else.

HOWEVER: I have noticed that the preaching priest will then often also take part in subsequent battles -- often, but not always. (I am unsure on this point; I *think* it is because I clicked him/her, had her move (with the rest by having selected "all"), then reclicked and reordered to preach.)
In other words, the stealthy priest who proselytized behind enemy lines a turn ago, is no longer sneaky and hence wants to fight. But I would like to have her/him continue to preach stealthily, even though the province is mine.
(I see no "thematic" reason for her job being vastly different just because the government has been changed.)
How can I do this?

thank you