Can you tell the difference? A study of human vs machine-translated subtitles

ABSTRACTWhile machine translation offers the potential for improved efficiency and cost savings, there are concerns about its accuracy and reliability compared to human translation. This study aims to investigate the potential of machine translation systems by analysing viewers’ ability to distinguish between subtitles generated by ChatGPT and those created by human translators in the English to Spanish language pair. The study involved 119 Translation and Interpreting degree students who watched eight subtitled clips containing puns, cultural references, humour, and irony: five of these were generated by ChatGPT and the remaining three were created by a human translator. Results indicate that participants were unable to accurately distinguish between ChatGPT-generated and human-generated subtitles, although lower quality subtitles were associated with non-human translation. Factors such as experience with ChatGPT and exposure to subtitled content were not significant predictors of the ability to identify ChatGPT-generated subtitles. However, year of study was found to be a significant predictor, suggesting that translation expertise is a crucial factor for non-human subtitle detection. Overall, these results have important implications for the use of machine translation in subtitle generation and the quality of subtitled content.KEYWORDS: Machine translationsubtitlingChatGPTtranslation qualityhuman vs machine translation Disclosure statementNo potential conflict of interest was reported by the author(s).Institutional review board statementThe study was conducted in accordance with the Declaration of Helsinki and following the regulations in force at the University of Alicante (Spain) for studies involving humans: https://web.ua.es/en/vr-investigacio/comite-etica/presentation.html.Informed consent statementInformed consent was obtained from all subjects involved in the study.Data availability statementThis study analysed publicly available datasets which can be found at http://hdl.handle.net/10045/133278.Additional informationNotes on contributorsJosé Ramón Calvo-FerrerJosé Ramón Calvo-Ferrer holds a PhD in Translation and Interpreting from the University of Alicante, Spain, where he has taught different modules on Translation, English and teacher training since 2008. His research interests lie in ICT in general and video games in particular for second language learning and translator training. He has published various books and papers on video games, translation and second language learning, and is a Visiting Lecturer at the Department of Language of Linguistics of the University of Essex, where he delivers lectures and workshops on video games and translation.