The Difference Between High Quality and High Popularity

August 14, 2013 Katharina Lochner

What a “like” really indicates

Do you take user ratings into consideration when booking a hotel or shopping online? Many people use ratings and comments on websites like Tripadvisor or Amazon for deciding whether to book or buy something. The reasoning behind this is that such ratings are reliable and accurate in a “wisdom of the crowd” like manner. However, recent research now questions this notion.

Lev Muchnik from the Hebrew University of Jerusalem and Sinan Aral and Sean J. Taylor from NYU Stern School of Business conducted an experimental study on a website on which users could give positive and negative ratings to and comment on articles on a social news aggregation website. They randomly assigned 101,281 comments submitted on the site to three different groups: up-vote, down-vote, or no vote (control).

It turned out that negative ratings were likely to be corrected. However, positively rated comments were more likely to get positive votes and received better final ratings on average than comments without a vote. This herding effect was dependent on the topic (only affected the topics politics, culture and society, and business, but not economics, IT, fun and general news) and the relationship between the voters (friends tended to up-vote their friends’ positive ratings and correct negative ones).

Thus, in positive ratings, it seems to be hard to distinguish between the influence of high quality of the item rated on the one hand and social influence bias on the other hand. A positive rating could thus result from the high quality of the item rated, but just as well from past positive votes. So, as an article in the NewYork Times on this studystates it: “Is something popular because it is actually good, or is it popular just because it is popular?”

The original article is in the August volume of the journal Science. There is also a short video explaining the study:

What does this mean? Negative ratings are likely to be accurate once there are a certain number of votes. However, when evaluating positive ratings, we need to be very critical and try to find more information. If there are written reviews of items, it might make sense to not only look at the positive, but also at the negative evaluations and compare both. Also take the relationship you have with the other voters into account. If they are people you like, be even more critical with your own evaluation. Be aware of the social influence bias!

About the Author

Katharina Lochner

Dr Katharina Lochner is the former research director for the cut-e Group which was acquired by Aon in 2017. Katharina is now a researcher and lecturer at the University of Applied Sciences Europe in Iserlohn, Germany. In her role at cut-e, she applied the research in organizational and work psychology to real-world assessment practice. She has a strong expertise in the construction and evaluation of online psychometric tools.

Follow on Linkedin Visit Website More Content by Katharina Lochner
Previous Article
How to Improve Self-Control
How to Improve Self-Control

Willpower is essential for making a career and for being well-off, but also for avoiding drug abuse or stay...

Time Pressure: the Best Condition for Successful Multi-tasking?
Time Pressure: the Best Condition for Successful Multi-tasking?

For many of our readers their working day is likely to be characterised by a variety of tasks, ranging from...

Subscribe to our talentNews