Combining machine translation and automated scoring in international large-scale assessments

Q1 Engineering Visualization in Engineering Pub Date : 2024-04-08 DOI:10.1186/s40536-024-00199-7
Ji Yoon Jung, Lillian Tyack, Matthias von Davier
{"title":"Combining machine translation and automated scoring in international large-scale assessments","authors":"Ji Yoon Jung, Lillian Tyack, Matthias von Davier","doi":"10.1186/s40536-024-00199-7","DOIUrl":null,"url":null,"abstract":"Artificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies. This study proposes combining state-of-the-art machine translations (i.e., Google Translate & ChatGPT) and artificial neural networks (ANNs) to mitigate two key concerns of human scoring: inconsistency and high expense. We applied AI-based automated scoring to multilingual student responses from eight countries and six different languages, using six constructed response items from TIMSS 2019. Automated scoring displayed comparable performance to human scoring, especially when the ANNs were trained and tested on ChatGPT-translated responses. Furthermore, psychometric characteristics derived from machine scores generally exhibited similarity to those obtained from human scores. These results can be considered as supportive evidence for the validity of automated scoring for survey assessments. This study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs.","PeriodicalId":37417,"journal":{"name":"Visualization in Engineering","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-04-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Visualization in Engineering","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1186/s40536-024-00199-7","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"Engineering","Score":null,"Total":0}
引用次数: 0

Abstract

Artificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies. This study proposes combining state-of-the-art machine translations (i.e., Google Translate & ChatGPT) and artificial neural networks (ANNs) to mitigate two key concerns of human scoring: inconsistency and high expense. We applied AI-based automated scoring to multilingual student responses from eight countries and six different languages, using six constructed response items from TIMSS 2019. Automated scoring displayed comparable performance to human scoring, especially when the ANNs were trained and tested on ChatGPT-translated responses. Furthermore, psychometric characteristics derived from machine scores generally exhibited similarity to those obtained from human scores. These results can be considered as supportive evidence for the validity of automated scoring for survey assessments. This study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs.

Abstract Image

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
在国际大规模评估中将机器翻译和自动评分相结合
人工智能(AI)正在迅速改变通信和技术驱动的内容创作,并越来越多地应用于教育领域。尽管取得了这些进步,但由于处理大量多语种答卷所带来的评分挑战,国际大规模测评(ILSA)中由人工智能驱动的自动评分在很大程度上仍未得到探索。然而,由于其低风险的性质,国际大规模测评是创新和探索新方法的理想场所。本研究建议结合最先进的机器翻译(即谷歌翻译和 ChatGPT)和人工神经网络(ANN),以减轻人工评分的两个主要问题:不一致性和高成本。我们将基于人工智能的自动评分应用于八个国家和六种不同语言的多语种学生答卷,并使用了 TIMSS 2019 的六个构建答卷项目。自动评分显示出与人工评分相当的性能,尤其是当人工智能网络在 ChatGPT 翻译的回答上进行训练和测试时。此外,从机器评分中得出的心理测量特征与从人工评分中得出的心理测量特征基本相似。这些结果可被视为调查评估自动评分有效性的支持性证据。本研究强调,自动评分与最新的机器翻译相结合,有望在 ILSA 中实现一致且节省资源的评分。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Visualization in Engineering
Visualization in Engineering Engineering-Engineering (miscellaneous)
CiteScore
8.60
自引率
0.00%
发文量
0
期刊介绍: Visualization in Engineering publishes original research results regarding visualization paradigms, models, technologies, and applications that contribute significantly to the advancement of engineering in all branches, including medical, biological, civil, architectural, mechanical, manufacturing, industrial, aerospace, and meteorological engineering and beyond. The journal solicits research papers with particular emphasis on essential research problems, innovative solutions, and rigorous validations.
期刊最新文献
The influence of religious attachment on intended political engagement among lower-secondary students Investigating item complexity as a source of cross-national DIF in TIMSS math and science Exploration of the linear and nonlinear relationships between learning strategies and mathematics achievement in South Korea using the nominal response model : PISA 2012 Combining machine translation and automated scoring in international large-scale assessments No substantive effects of school socioeconomic composition on student achievement in Australia: a response to Sciffer, Perry and McConney
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1