Be your own doctor: Temperature scaling self-knowledge distillation for medical image classification

Abstract

Self-knowledge distillation (self-KD), which uses the student network as the teacher model, allows the model to learn knowledge by itself. It has been widely studied in various medical image tasks for constructing lightweight models to alleviate the limitations of computing resources. However, existing self-KD methods use a single temperature for distillation, ignoring the effect of temperature on different classes. In this paper, we investigate the effects of target class temperature and non-target class temperature on the performance of self-KD. Based on the above study, a temperature scaling self-knowledge distillation (TSS-KD) model is proposed, which can better balance the target class knowledge and non-target class knowledge. By adjusting the temperature scaling of different classes, the model can learn better representations by distilling the well-proportioned features. To make the network focus more on the local lesions of medical images, a regional gamma augmentation (RGA) method is proposed, which provides stronger perturbations to the same sample to generate more differentiated features. By self-regularizing the consistency of these features, the model can learn more local knowledge. To evaluate the effectiveness of the proposed method, extensive experiments are conducted on nine medical image classification tasks of eight public datasets. Experimental results show that the proposed method outperforms state-of-the-art self-KD models and has strong generality. The code is available at https://github.com/JeaneyLau/TSS-KD.