Beyond BLEU: Repurposing neural-based metrics to assess interlingual interpreting in tertiary-level language learning settings

Abstract

Recent years have seen a revival of using translation and interpreting (T&I) as a pedagogical and assessment tool to enhance language learning. This growing usage contributes to an increasing amount of learner-generated T&I data, creating a strong demand for assessment. To alleviate this issue, researchers have proposed repurposing machine translation (MT) evaluation metrics to automatically assess human-generated T&I. In this article, we report on the first large-scale study in which we leveraged sophisticated neural-based MT evaluation metrics for automatically assessing English-Chinese interpreting, using a database called Interpreting Quality Evaluation Corpus. To evaluate the efficacy of neural-based metrics, we correlated them with human benchmark scores. Because of the unique data structure, we conducted an internal meta-analysis of correlation coefficients to examine the overall machine-human correlation, and further performed meta-regression to identify potential significant moderators. We find that: a) the overall meta-synthesized correlations were fairly strong: r = .652 and r_s = .631; b) the type of neural-based metrics was a significant moderator, with BLEURT-20 registering the highest correlations (r = .738, r_s = .700); and c) the level of human rater reliability was also a significant moderator. We discussed these findings and their implications for T&I assessment in higher education.