An Experimental Study to Understand User Experience and Perception Bias Occurred by Fact-checking Messages

Abstract

Fact-checking has become the de facto solution for fighting fake news online. This research brings attention to the unexpected and diminished effect of fact-checking due to cognitive biases. We experimented (66,870 decisions) comparing the change in users’ stance toward unproven claims before and after being presented with a hypothetical fact-checked condition. We found that, first, the claims tagged with the ‘Lack of Evidence’ label are recognized similarly as false information unlike other borderline labels, indicating the presence of uncertainty-aversion bias in response to insufficient information. Second, users who initially show disapproval toward a claim are less likely to correct their views later than those who initially approve of the same claim when opposite fact-checking labels are shown — an indication of disapproval bias. Finally, user interviews revealed that users are more likely to share claims with Divided Evidence than those with Lack of Evidence among borderline messages, reaffirming the presence of uncertainty-aversion bias. On average, we confirm that fact-checking helps users correct their views and reduces the circulation of falsehoods by leading them to abandon extreme views. Simultaneously, the presence of two biases reveals that fact-checking does not always elicit the desired user experience and that the outcome varies by the design of fact-checking messages and people’s initial view. These new observations have direct implications for multiple stakeholders, including platforms, policy-makers, and online users.