Does User Choice of Device Impact the Results of Online Surveys?

Abstract

This article investigates how and to what extent the data collected, the dropout rate, and the completion time in online surveys is influenced by the device used to fill out the questionnaire. To that end, an extensive online study with N=1493 was carried out. To address difficulties associated with the use of devices with smaller displays, different layout variants aimed at optimizing questionnaire usability for smartphones were developed and analyzed. Completion time, drop-out rate, and response patterns were compared across different display sizes and layout variants. Results show significantly lower completion times and drop-out rates when the questionnaire was answered on a larger display. Also, different answering patterns emerged among participants using mobile devices. Likewise, the study revealed effects of different questionnaire layouts. The authors discuss implications for the design of online questionnaire in order to obtain reliable data from online surveys.