Has to hold the castle, I believe - simply besieging a victory point province doesn't give the VP, just as the person inside the fort keeps the magic sites.
And, I'm trying to drive him out of a VP province this turn, but he's also likely to take an empty aquatic VP province.
We'll see.