Bias in job analysis survey ratings attributed to order effects

Abstract

When respondents complete long surveys, evidence suggests that they may change their response behavior as they become fatigued. The present study examines question order effects in a block-randomized job analysis survey that collected task ratings (importance, frequency, and needed at certification) for a certification program. Analyses showed that average task ratings were lower for tasks presented later in the survey for both complete and partial responses, though the effect of question ordering was stronger for non-completers in two rating categories (importance and needed at certification). Additionally, rater variance was lower for questions that appeared later in the survey. The results highlight the value of using randomization to mitigate order effects on survey response quality.