Adaptive Cross-Modal Experts Network with Uncertainty-Driven Fusion for Vision–Language Navigation

Abstract

Vision-and-Language Navigation (VLN) enables an agent to autonomously navigate in real-world environments based on language instructions to reach specified destinations and accurately locate relevant targets. Although significant progress has been made in recent years, two major limitations remain: (1) Existing methods lack flexibility and diversity in processing multimodal information and cannot dynamically adjust to different input features. (2) Current fixed fusion strategies fail to dynamically adapt to varying data quality in open environments, insufficiently leveraging multi-scale features and handling complex nonlinear relationships. In this paper, an adaptive cross-modal experts network (ACME) with uncertainty-driven fusion is proposed to address these issues. The adaptive cross-modal experts module dynamically selects the most suitable expert network based on the input features, enhancing information processing diversity and flexibility. Additionally, the uncertainty-driven fusion module balances coarse-grained and fine-grained information by calculating their confidences and dynamically adjusting the fusion weights. Comprehensive experiments on the R2R, SOON, and REVERIE datasets demonstrate that our approach significantly outperforms existing VLN approaches.