The effectiveness of warning statements in reducing careless responding in crowdsourced online surveys.

Abstract

Carelessness or insufficient effort responding is a widespread problem in online research, with estimates ranging from 3% to almost 50% of participants in online surveys being inattentive. While detecting carelessness has been subject to multiple studies, the factors that reduce or prevent carelessness are not as well understood. Initial evidence suggests that warning statements prior to study participation may reduce carelessness, but there is a lack of conclusive high-powered studies. This preregistered randomized controlled experiment aimed to test the effectiveness of a warning statement and an improved implementation of a warning statement in reducing participant inattention. A study with 812 participants recruited on Amazon Mechanical Turk was conducted. Results suggest that presenting a warning statement is not effective in reducing carelessness. However, requiring participants to actively type the warning statement statistically significantly reduced carelessness as measured with self-reported diligence, even-odd consistency, psychometric synonyms and antonyms, and individual response variability. The active warning statements also led to statistically significantly more attrition and potentially deterred those who were likely to be careless from even participating in this study. We show that the current standard practice of implementing warning statements is ineffective and novel methods to prevent and deter carelessness are needed.