Yelp’s Review Filter Sometimes Does More Harm than Good

Filed in Local Search by Matt McGee on April 25, 2012 3 Comments

yelp-logoYelp’s review filter is biased against first-time reviewers. This is no secret. It’s why some previous studies have shown that only nine percent of Yelp users have published just one review.

People like that — hit-and-run reviewers — aren’t welcome at Yelp, no matter what they have to say.

That’s all fine and good when Yelp’s review filter suppresses spammy, untrue, 5-star reviews of businesses that don’t deserve it. But, as Justin Vincent wrote yesterday, it’s not all fine and good when the filter suppresses legitimate one-star reviews.

Vincent says that he chose a moving company based on the preponderance of 5-star reviews it had. But, after a terrible experience that cost him $2,000, he found 10 negative reviews that Yelp had filtered because they came from non-power users.

Your algorithm typically hides entries by people who only post one review and who don’t otherwise engage in Yelp. Your assumption is that if a user only posts one review, posts no comments, has no friends etc. then most likely they are fake and trying to game the system.

Let’s call this “Assumption X”.

In the case of the company that I mention above (the one that ripped me off) Assumption X is exactly wrong at least 10 out of 14 times. Just to be clear, 10 honest one star reviews have been hidden from public view. That’s a 71% false positive hit rate.

So why did Yelp get it wrong 10 times?

In each case the one star review was left by someone who would never normally leave a review… they were simply so outraged that they were motivated to signup to Yelp and try to warn others how bad this company is. None of them ever used Yelp again. Furthermore, they didn’t have the knowledge or inclination to try to make their Yelp profile look acceptable to Yelp’s automated suppression systems.

Whether you love Yelp or hate it (and I don’t have a strong opinion one way or another, so please, fanboys, save your venom for someone else), that’s a blog post worth reading and thinking about. I’m not sure I agree with the solution that Vincent proposes, but I do agree that Yelp has to come up with some improvements that allow real reviews to show up on a business listing, whether they come from active Yelpers or not.

Comments (3)

Trackback URL | Comments RSS Feed

  1. MiriamEllis says:

    Thanks for linking to this, Matt. Worth reading. Hope you and your family are doing well!

  2. Kevin McCusker says:

    Used to use it just to read reviews. Then started reviewing. 4 in (2 1 stars one 5 star and a 4 star) and they have all been fiktered even though completely legitimate, within guidelines, and detailed. Its exactly what makes me want to not review again. Just rather annoying to new users who want to start up.

    • Matt McGee says:

      Your story is not uncommon, Kevin. I started reviewing on Yelp before the review filter existed, or maybe before it was as aggressive as it is now. My first review was a pretty scathing piece about a terrible hotel, and I’m sure it wouldn’t survive the filter today if I was just signing up and trying to post it.

Leave a Reply

Your email address will not be published. Required fields are marked *