Randomizing survey question order vs. grouping questions by construct: an empirical test of the impact on apparent reliabilities and links to related constructs

Abstract

There has been a quiet debate about the proper way to design a questionnaire when multiple questions measure each construct. Two general arguments favor intermixing the questions randomly with those of other constructs; one argument favors grouping together questions that measure a single construct. This study tests these arguments using two different versions of a Web quality instrument. One version has questions grouped and labeled, and the other has questions randomly intermixed. Six hundred and three undergraduate students filled out questionnaires, half in each condition. Several different analyses suggest that the grouped question treatment has higher calculated Cronbach's alpha reliabilities than intermixed questions, but is actually less reliable.