Multimodal Machine Translation Enhancement by Fusing Multimodal-attention and Fine-grained Image Features

Abstract

With recent development of the multimodal machine translation (MMT) network architectures, recurrent models have effectively been replaced by attention mechanism and the translation results have been enhanced with the assistance of fine-grained image information. Although attention is a powerful and ubiquitous mechanism, different number of attention heads and granularity image features aligned by attention have an impact on the quality of multimodal machine translation. In order to address above problems, this paper proposes a multimodal machine translation enhancement by fusing multimodal-attention and fine-grained image features method which builds some submodels by introducing different granularity of image features to the multimodal-attention mechanism with different number of heads. Moreover, these sub-models are randomly fused and fusion models are obtained. The experimental results on the Multi30k dataset that the pruned attention heads lead to the improvement of translation results. Finally, our fusion model obtained the best results according to the automatic evaluation metrics BLEU compared with sub-models and some baselines.