Do choice tasks and rating scales elicit the same judgments?

Abstract

Discrete choice (DCE) and rating scale experiments (RSE) are commonly applied procedures for eliciting preference judgments in a plethora of applied settings such as consumer choices, health care, and transport economics. An almost universal assumption is that actual “ground truth” preferences do not depend on which elicitation procedure is used. It is usually not possible to test this assumption, because typical studies feature response options for which there is no objectively correct response. To make progress on testing this assumption, we conducted a perceptual discrimination experiment where response options varied on a single attribute – stimulus saturation level – with a known objectively correct response. We had the same participants complete both a choice task (CT) and rating scale (RS) version of the experiment, allowing a direct examination of the assumption of a common representation. Our CT featured many characteristics that define a DCE, however, in order to have a known objectively correct response, it also differed in a few important ways. To test the assumption of a common representation, we developed a cognitive model with a response mechanism for both CT and RS. This enabled us to compare a model version that featured one shared latent stimulus representation across CT and RS versus a version which featured separate representations. Our results support the assumption that a single internal state supports both CT and RS responses, and also suggest that the CT method might provide more sensitive measurement of internal states than the RS method.