Overcoming language barriers via machine translation with sparse Mixture-of-Experts fusion of large language models

Abstract

Large language models (LLMs) hold great promise for cross-lingual applications to power machine translation (MT) systems. However, directly fine-tuning LLMs on parallel data risks catastrophic forgetting and lacks explainability in cross-lingual knowledge transfer. In this paper, we introduce MoE-LLM, a novel fusion framework that enhances the multilingual translation abilities of LLMs by incorporating sparse Mixture-of-Experts (MoEs) components via hybrid transfer learning. MoE-LLM freezes the LLM parameters, mitigating forgetting, and introduces specialized translation experts within the MoEs modules. Our hybrid initialization strategy further bridges the representation gap by warm-starting MoE parameters using LLM representations. We evaluated MoE-LLM on 10 translation directions across 6 languages using the WMT benchmark. Compared with directly fine-tuning LLMs, MoE-LLM significantly improved translation quality, achieving gains of up to 2.5 BLEU points, with at least some improvement in zero-shot translation scenarios and surpassing other strong baselines like Adapter and LoRA-F. Our ablation studies highlight the effectiveness of the cascaded fusion strategy and the mixed initialization approach for optimal performance. MoE-LLM offers an effective and explainable solution for adapting pre-trained LLMs to multilingual machine translation, with particular benefits in low-resource and zero-shot scenarios.