That does often happen for one spy...
Try sending in about 4 or 5 at once, though. They are cheap enough that this isn't really cost-prohibitive. Tell me what happens then.
I think unrest helps to hide your spies...but that's just my supposition. And spies are better hidden than scouts with their stealth +20.