Search results pages are hard to design. When crafting these areas, understanding how a user narrows down a large field of choices is incredibly important, and isn’t always inherently obvious. As an example, a user trying to narrow down choices on a travel site may use several factors in order to weed the choices down to a manageable set: price, location, rating, amenities, etc.
Lately, as more websites offer users the ability to rate content (books, recipes, hotels, etc.), sorting and narrowing a set of search results by rating is becoming an increasingly popular method employed by users. It makes sense, as it harnesses the crowdsourcing nature of the web and allows users to cull out the products that the community has decided are good.
However, how these ratings are sorted is handled differently from website to website. Recently, I had been searching for cookbooks on Amazon. I sorted by rating to cull out what I thought were books with the best community reputation. Below is a screenshot of a couple of these results:
Notice this: these products are shown in order of decreasing rating, and the book with one 5-star rating is positioned above the book with 159 ratings, which have resulted in a 4.5 star average.
Now, technically, this is a correct way of sorting. Yes, 5 stars are better than 4.5. However, does one 5-star rating give as accurate a quality indication as an average of 4.5 stars over 159 reviews? Probably not. The 5-star book could be given a 1-star rating by the next user, resulting in it dropping to a much lower average. This kind of volatility means that the ability for the user to make a culling decision based strictly off rating (and many users do use one factor like this to get a list down to a manageable size) is unreliable.
So, what’s the solution? I think it’s a calculated score of some sort. It may be as simple as total number of reviews/average score (159 reviews/4.5 stars). I’m not sure if this holds up on all ends of the scale - I’ll let the math contingent chime in with their suggestions for a better system.
As a final note, the recipe site Recipezaar does appear to use some sort of calculated method for sorting by rating. Note the following results:
Aha. This time, the result with a slightly lower star rating, but more total reviews, is ranked higher than the result with a higher star rating over fewer reviews.
What do you think? Is Recipezaar onto something? Is Amazon wrong?
I definitely like Recipezaar ranking more. Mainly because it takes number of reviews into account.
One of the best ranking mechanisms I’ve ever seen can be found on kde-apps.org and kde-look.org websites. They use percentages but number of votes does matter.