Inconsistent rating scales decrease social influence bias and enhance crowd wisdom

Abstract

Online ratings for products and services pervade society. Research on the wisdom-of-the-crowd phenomenon suggests that the average of individuals' ratings should reflect true underlying quality. Online platforms that collect ratings, however, often do so while displaying the crowd's average rating, creating the potential for social influence. This is problematic because the wisdom-of-the-crowd effect relies on the aggregation of independent evaluative judgments. How can platforms limit social influence bias and enhance crowd wisdom? We argue that the structure of the rating scales used to record individual evaluations can alter the degree of social influence bias in online rating platforms. Through an analysis of rating websites with over 4.4 million ratings from over 60,000 evaluators and an online experiment (200 participants), we show that by varying the rating scales that record individual evaluations, platforms can decrease the presence of social influence, better maintaining the independent judgments that are necessary for unbiased crowd wisdom. Understanding how to limit social influence bias in online ratings stands to improve quality judgments across large swaths of economic and social life.