Transformer-based text similarity and second language proficiency: A case of written production by learners of Korean

Abstract

The present study applies two transformer models (BERT; GPT-2) to analyse argumentative essays produced by two first-language groups (Czech; English) of second-language learners of Korean and investigates how informative similarity scores of learner writing obtained by these models explain general language proficiency in Korean. Results show three major aspects on model performance. First, the relationships between the similarity scores and the proficiency scores differ from the tendencies between the human rating scores and the proficiency scores. Second, the degree to which the similarity scores obtained by each model explain the proficiency scores is asymmetric and idiosyncratic. Third, the performance of the two models is affected by learners’ native language and essay topic. These findings invite the need for researchers and educators to pay attention to how computational algorithms operate, together with learner language characteristics and language-specific properties of the target language, in utilising Natural Language Processing methods and techniques for their research or instructional purposes.