Examining Manual and Automatic MT Evaluation: A Grey Relational Analysis for Chinese-Portuguese Translation Quality

Abstract

This study investigates the relationship between manual and automatic machine translation evaluation methodologies by employing Grey Correlation Analysis (GRA) to assess the correlation between Chinese-Portuguese machine translation outputs’ BLEU scores and human evaluation scores based on a proposed evaluation index system. The research aims to provide insights into the factors impacting machine translation quality and the most relevant linguistic dimensions in translation evaluation. The findings reveal that “usability” and “adequacy” exhibit the highest correlation with BLEU scores, while “Semantics” ranks highest among the ten manual evaluation indicators, when correlated with the aggregated human evaluation results, followed by “Correction” (correct information) and “Omission and/or Addition”. The findings contribute to the field of machine translation evaluation by illuminating the complex relationship between manual and automatic evaluation techniques and guiding future improvements in machine translation systems and evaluation methodologies.