Difficulty level-based knowledge distillation

Abstract

Knowledge distillation (KD) enables a simple model (student model) to perform as a complex model (teacher model) by distilling the knowledge from a pre-trained teacher model. Existing soft-label distillation methods often use a fixed temperature value in the softmax function to prevent overconfidence in the distillation process. However, this approach can lead to the suppression of important ‘dark knowledge’ for non-target classes in difficult samples, while also over-smoothing the confidence values for easier samples. To address this issue, we propose a novel approach called difficulty level-based knowledge distillation (DLKD), which considers the difficulty level of each sample to distill refined knowledge with high or low confidence, depending on the sample’s complexity. Our method calculates the difficulty level based on the Euclidean distance between the teacher model’s predictions and the pruned teacher model’s predictions. Experimental results demonstrate that our DLKD method outperforms state-of-the-art methods on challenging samples, including those with noisy labels or augmented data, achieving superior results on CIFAR-100, FGVR, and ImageNet datasets for image classification.