Asking about the Worst First: An Examination of Contextual Effects in Factorial Vignettes

Abstract

Contextual effects refer to the process by which responses given to survey questions can be affected by question order. Generally, contextual effects harm data measurement validity by introducing bias and increasing measurement error; the risk is that responses to a survey’s later questions are partly affected not only by the substance of the question but also by the preceding questions. Two opposite effects are possible: a carryover effect refers to the assimilation of later questions into those previously asked, and a backfire effect refers to the contrasting of earlier and later questions. In the case where a stereotype is activated in earlier questions of a survey, the previous literature suggests a carryover effect is more likely. The present study tests whether this is also the case in factorial vignette research by examining the influence of first presenting a vignette that corresponds more closely to a stereotypical view of sexual abuse. Results indicate a backfire effect, pointing to the distinctively different way in which vignette scenarios activate stereotypes compared to general survey questions. The results also highlight the need for researchers to control for contextual ordering effects when modeling factorial vignette data.