You should report them.And there's the rub.
What do you do when someone's not breaking the rules, just reducing the enjoyment?
Especially when that person is reducing it for the many, who all have to suffer.
They could do like you suggest - leave. But who does that serve? Certainly not the enjoyment of that forum, least of all that actual forum itself.
So what do you do if there's someone who's not actually breaking the rules but acting like a vampire and sucking the life out of it?
I suppose I'll have to leave that as a rhetorical question.
A certain gentleman who posted a lot and rarely broke rules but annoyed a lot of people was finally banned for the good of the forum. Members who were leaving because of him, or wanted to, had quiet words with one of the moderators. It would have been better and quicker if they had used that Report function. If people just ignore, or wander off, or only mention to a mod as they go why they are not happy, it is very hard to make a decision when the offending party isn't really breaking rules.