Knowledge Distillation SegFormer-Based Network for RGB-T Semantic Segmentation

Abstract

Deep-learning-based semantic segmentation has received increasing research attention in recent years. However, owing to complex architectures, existing approaches have failed to achieve high accuracies in real-time applications. In this article, a novel knowledge distillation (KD) SegFormer-based network, called KDSNet-S*, is proposed to explore the tradeoff between accuracy and efficiency. Specifically, a structured KD scheme is designed to transfer the rich advanced features of a teacher network (KDSNet-T) to a student network (KDSNet-S). Thereafter, the KDSNet-S network learns the precise segmentation ability of the KDSNet-T network. Additionally, a multifield perceptual fusion model is proposed to learn more integrated features for a single modality and obtain discriminative and comprehensive feature representations. Furthermore, a high-level feature integration module is introduced to refine multimodality high-level features. Finally, multilevel features are fused, and a label-decoupling-based three-stream decoder that decomposes the original semantic segmentation map into center and contour diffusion maps for different supervision tasks is introduced. Experimental results on two public red-green–blue-thermal semantic segmentation datasets indicate the superiority of KDSNet-S* over compared state-of-the-art methods. The KDSNet-S* reduces parameters and floating-point operations per second by 91.1% and 81.9%, respectively, compared with the KDSNet-T. The source codes and results will be available at https://github.com/purple-ting/KDSNet.