Deeper and Broader Multimodal Fusion: Cascaded Forest-of-Experts for Land Cover Classification

Abstract

Multimodal land cover classification (LCC) of optical and SAR images has become a research hotspot. However, there are still two unsolved problems: the lack of a deep fusion mechanism and the neglect of the diversity of multimodal features. Inspired by ensemble learning, this letter proposes the cascaded multimodal forest-of-experts (CM2FEs) for deeper and broader fusion to further improve the performance of LCC. The proposed method first establishes the expert tree, then combines multiple trees at the same level into a forest, and finally forms a cascaded forest across different levels. Specifically, the novel designs include three points: 1) the multimodal expert tree is built based on linear projection and dynamic routing, with multiple layers of experts; it can acquire more discriminative multimodal features through deeper fusion; 2) the cascaded forest is formed by combining expert trees at the same level and different levels, which can effectively ensemble the knowledge learned by different trees; it can generate more diverse multimodal features through broader fusion; and 3) two expert exchange strategies are proposed to transfer knowledge between different trees and further optimize the feature fusion effect. Experiments show that the proposed method performs better than existing methods, and the mean IoU (mIoU) has been improved by at least 1.60%–3.25%.