Content Analysis of Test Taker Responses on an AP Japanese Computer-Simulated Conversation Test: A Mixed Methods Approach for A Validity Argument

ABSTRACT The present study is part of a larger mixed methods project that investigated the speaking section of the Advanced Placement (AP) Japanese Language and Culture Exam. It investigated assumptions for the evaluation inference through a content analysis of test taker responses. Results of the content analysis were integrated with those of a many-facet Rasch analysis of the same speech data. This study found that most information-seeking prompts elicited a good sized ratable speech sample with relevant content, and the rating criteria seemed to fit with the nature of the interaction. Therefore, information-seeking prompts generally provided appropriate evidence of test takers’ ability. In contrast, non-information-seeking prompts such as requests and expressive prompts tended to have issues with eliciting a good sized ratable speech sample with relevant content, and their response expectations realized in the rating criteria did not fit with the nature of the interaction. Thus, non-information-seeking prompts showed greater potential of becoming sources of measurement error with the current test design. This article discusses possible solutions to increase the validity of the evaluation inference. Findings from the present study would be useful for future test development of computer-based L2 tests that aim to assess interpersonal communication skills.