Measurement and evaluation of participant response consistency in Case 1 Best-Worst-Scaling (BWS) in food consumer science

Abstract

Best-Worst Scaling (BWS) is well suited to survey research, and since this way of collecting data is gaining popularity, it is likely that applications of BWS, especially online, will increase. Despite the many advantages of online surveys, there is a growing awareness that particular attention needs to be given to data quality including the identification and elimination of ’bad respondents’. Post hoc data cleaning is the focus of the present research, which in the context of Case 1 BWS (object case) was directed to two indices of participant response consistency – root likelihood (RLH) and normalised error variance (ErrVarNorm), where the latter was newly developed. Across 18 studies in food consumer science, the two indices are applied, compared and evaluated. It is possible to apply both indices to the data, but if only a single index is to be used, the ErrVarNorm index is recommended because it is easy to calculate, directly measures response consistency and is logically coherent. ErrVarNorm ranges from 0 to 1, where higher values indicate greater response consistency. When excluding participants based on ErrVarNorm < 0.3, between 0 % and 12.4 % of participants were excluded (mean = 5.75 %), while ErrVarNorm < 0.5 led to between 0 % and 23.8 % of participants being excluded (mean = 13.1 %). Excluded participants were more likely to be men and below the age of 45 years old. The impact on study conclusions when excluding participants based on ErrVarNorm < 0.5 were illustrated for three studies in support of the importance of data cleaning.