Oh, I never said it was a perfect system. And certainly, the permanency of a rating is one of them you correctly pointed out.
All I can say is that Ratings (like statistics) average out over time on the predication that most people aren't vindictive and are generally objective.
Certainly, I've looked at a few Ratings that were low and wondered why so. Doesn't mean I "write" those people off, especially if I've never seen any reason for it. I understand your point, though, and I see how some might automatically assume someone is a jerk due to a rating of 3 (out of, say, just 2 votes). On the other hand, if one sees a rating of 1.2 (out of 15 votes) for someone, it seems more obvious it's not just the opinion of just a few people who got upset on a bad day.
To date, I can say the Ratings I have given people have not needed to be changed (due to some wild fluctuation in their actions).
And even though we (seem to) disagree a lot, you rate a 5-star with me (so THERE!
).