Assessing Whisper automatic speech recognition and WER scoring for elicited imitation: Steps toward automation

Abstract

Elicited imitation (EI) has received considerable attention in the field of SLA as a psycholinguistic method for oral proficiency assessment. However, EI tests are perhaps underutilized because of the need for time-consuming scoring by hand. A clear solution to this is computer automation, which requires two key components to work in parallel: (1) an accurate automatic speech recognition (ASR) system to transcribe EI response speech into text, and (2) a computational scoring metric to assess the resulting text. In this paper, we test the feasibility of automating EI scoring with the open-source Whisper ASR system and the Word Error Rate (WER) scoring metric which measures deviation from the original prompt. 30 Japanese L1 participants took a 30-item test, resulting in 900 English L2 learner EI sentence responses which were transcribed by the two authors and Whisper ASR. The intraclass correlation coefficient (ICC) between the error rates of the human raters and Whisper across all items was ICC = 0.929, 95 % CI [0.921, 0.936] indicating very strong alignment. We then compared automated test scores using WER to those done manually by human raters using a traditional ordinal-scale scoring method developed by Ortega et al. (2002) and found a robust correlation of r = 0.969, 95 % CI [0.935, 0.985] across overall participant scores. These findings show that the combination of the Whisper ASR system and the WER scoring metric result in EI test scores that align almost perfectly with and are arguably an improvement on currently accepted methods.