Joint Classification of Hyperspectral Images and LiDAR Data Based on Candidate Pseudo Labels Pruning and Dual Mixture of Experts

Abstract

Hyperspectral images (HSIs) contain rich spatial and spectral information, while light detection and ranging (LiDAR) data can provide elevation details. Effectively fusing HSI and LiDAR data can help achieve more accurate classification results. However, the joint classification of HSI and LiDAR data still faces several challenges, such as the redundancy of HSI spectral bands, the limitations of singular multimodal data fusion strategy, and the high cost of pixelwise labeling in remote sensing images. To tackle these challenges, we propose a classification method based on candidate pseudo labels pruning and dual mixture of experts (CPLP-DMoEs). First, we employ the multihead mixture of bands (MMoBs) to perform diverse, dense mixing of spectral bands, thereby alleviating the issue of high similarity between adjacent bands. Then, to overcome the limitations of single fusion strategies, we design a mixture of multimodal fusion expert (MoMFE) mechanism, which selects and mixes multiple fusion experts (FEs) to achieve diverse feature fusion of HSI and LiDAR data. Next, we introduce information entropy to balance the selection of FEs. Finally, facing the challenge of limited labeled samples, we propose a candidate pseudo labels pruning (CPLP)-based semi-supervised learning method. CPLP can prune the candidate pseudo label set from both intrasample and intersample perspectives to obtain more reliable pseudo labels, thereby facilitating the learning of a more accurate classification model. The experimental results on three datasets, including Houston 2013, MUUFL, and Augsburg, validate the effectiveness of the proposed method.