Evaluation of Transformer model performance on a set of language pairs by varying standard parameters

Abstract

In this paper, we have performed machine translation using modified Transformer model on TED Talk Dataset and from this we have selected four language pairs (English-Portuguese, Russian-English, French-Portuguese, Turkish-English) for testing. We have evaluated and analyzed the machine translation model on critical criteria's such as dataset size, batch size and training time over all four language pairs and have used BLEU scoring to generalize the results. We have searched for general trends and have analyzed the impacts of these variations on the scoring of the translation upon variation of different parameters, this analysis has been performed on each language pairs as well as have been compared with the results of other language pairs. From the results we found that with increase in dataset size, batch size and training time the BLEU score increases. The English - Portuguese pair achieves highest BLEU score of 23.2 from all language pairs and we get lowest BLEU score of 19.1 from the French - Portuguese pair.