Enhancing knowledge distillation for semantic segmentation through text-assisted modular plugins

Abstract

Compared with other model compression methods, such as pruning and quantization, knowledge distillation offers superior compatibility and flexibility. Current knowledge distillation (KD) methods for semantic segmentation predominantly guide the student model to replicate the structured information of the teacher model solely through image data. However, these approaches often overlook the potential benefits of incorporating auxiliary modalities, such as textual information, into the distillation process, thereby failing to effectively bridge the gap between the student and teacher models. This paper introduces a novel text-assisted distillation methodology. Leveraging the framework of Contrastive Language-Image Pretraining (CLIP), we propose two modular plugins: the Text-Channel Distillation module and the Text-Region Distillation module, designed to integrate textual priors into the distillation process. These modules serve as a bridge between the student and teacher models, enhancing the emulation of teacher networks by student models. Characterized by their simplicity, versatility, and seamless integration with existing knowledge distillation frameworks, these modules facilitate improved performance. Experimental evaluations conducted on the Cityscapes, Pascal VOC, and CamVid datasets demonstrate that augmenting state-of-the-art distillation techniques with these plug-and-play modules yields significant improvements in distillation effectiveness.