[Kamamura]

You are mistaken there. Peasants were allowed to take refuge, because they were not just mouths to feed - they helped the defense by extinguishing fires, and doing various support works. Moreover, it was the promise of security that made the feudal life less unbearable - a feudal lord that would not protect his subject in times would face desertion/revolts/loss of prestige. What's more, peasants, even if not free subjects, were source of income and wealth and well worth protecting.