NoRD: A framework for noise-resilient self-distillation through relative supervision

Abstract

Knowledge distillation (KD) has become a pivotal technique in deep learning, facilitating model compression and regularization by transferring knowledge from one neural network to another, enhancing its capabilities for downstream tasks such as classification. However, real-world datasets often suffer from noisy label problems, significantly hindering neural network learning in supervised tasks. Recent advancements in KD aim to improve noise-robustness and regularization in deep neural networks through different learning paradigms. Yet, prevalent approaches often exhibit noise-prone behaviors as the student network heavily relies on the teacher’s learning. To address this challenge, we propose a robust knowledge transfer method, NoRD: a Noise-Resilient Self-Distillation framework. This approach leverages relative self-supervision combined with decision matching to minimize noise susceptibility during the knowledge transfer process. Our study evaluates this technique on CIFAR-10, CIFAR-100, and MNIST datasets with synthetic label noise. Results showcase that our method achieves 8-10% higher test accuracy compared to state-of-the-art noise-robust loss functions at noise rates exceeding 50%, surpassing well-known KD methods by 4-5% in top-1 test accuracy. The code is available at https://github.com/philsaurabh/NoRD_Applied-Intelligence.