Multidimensional knowledge distillation for multimodal scene classification of remote sensing images

The advancement of deep learning technology has significantly improved the performance of remote sensing image (RSI) scene classification. However, it is important to note that most RSI scene classification models heavily depend on complex structures, resulting in high computational requirements and substantial costs. This study addresses this issue by utilizing a state-of-the-art model compression technique known as knowledge distillation (KD). The objective of KD is to transfer extensive knowledge from an excellent teacher model to a lightweight student model. While existing models focus on guiding the student network to learn specific stage or scale features from the teacher network, they lack comprehensiveness. To enhance the model's feature representation capability in complex scenarios, this study proposes a multidimensional KD approach (MKD). MKD enables the student network (MKD-S) to learn the feature representation capability of the teacher network (MKD-T) at each stage through a hybrid KD method. Specifically, the encoder incorporates a local-global KD mechanism to capture both low-level local information and high-level global information based on feature differences. Moreover, the fusion stage introduces inter-layer relationship KD and intra-layer feature KD to account for the dependencies between intermediate features within the MKD-S and MKD-T models. Additionally, the discrete wavelet transform, known for its ability to capture frequency domain and time domain features, is applied in the decoding stage of the MKD-T. This integration of decoding features across layers enables the completion of the knowledge response in the MKD-S. Experimental results demonstrate the effectiveness of our MKD on two benchmark datasets: Vaihingen and Potsdam.