Rethinking Think-Alouds: The Often-Problematic Collection of Response Process Data

ABSTRACT The objective of this paper is to comment on the think-aloud methods presented in the three papers included in this special issue. The commentary offered stems from the author’s own psychological investigations of unobservable information processes and the conditions under which the most defensible claims can be advanced. The structure of this commentary is as follows: First, the objective of think-alouds in light of test development and validation goals are considered for each of the three papers in the volume. Second, the response processes (psychological constructs) described in the three studies are assessed vis à vis think-aloud methods. Third, the methodological details that are essential to properly evaluate response processing data for educational assessment goals are elaborated. Fourth, the possible impasse of using a psychological technique to collect psychological data about non-psychological content forms the basis of the commentary’s conclusion.