Question-Order Effect in the Study of Satisfaction with Democracy: Lessons from Three Split-Ballot Experiments

Abstract

This study examines question-order effects in measuring satisfaction with democracy (SWD). Particularly, the authors are interested in whether the relative position of the question regarding satisfaction with the state of the economy (SWE) in the questionnaire affects responses to the SWD item. The authors conducted three independent split-ballot experiments in Hungary between March 2021 and May 2022. They report a significant and substantial negative priming effect that possibly leads to a systematic underestimation of SWD. Importantly, the authors find no question-order effect in the measurement of SWE. The analysis further reveals a contrast effect: when the SWD question is primed, the difference between SWE and SWD means increases. The authors’ final recommendation is that researchers either put the SWD question before the SWE item to avoid question-order bias or randomize question order. These findings should assist future data collection efforts (comparative or single-country studies) in developing and integrating a battery of satisfaction items into questionnaires and help users assess data quality.