A gradual approach to knowledge distillation in deep supervised hashing for large-scale image retrieval

Abstract

Deep learning-based hashing methods have emerged as superior techniques for large-scale image retrieval, surpassing non-deep and unsupervised algorithms. However, most hashing models do not consider memory usage and computational costs, which hinders their use on resource-constrained devices. This paper proposes an Optimized Knowledge Distillation (OKD) approach for training compact deep supervised hashing models to address this issue. OKD utilizes a unique growing teacher-student training strategy where an evolving teacher continuously imparts enriched knowledge to the student. The teacher and student networks are divided into blocks, with auxiliary training modules placed between corresponding blocks. These modules extract knowledge from intermediate layers to capture multifaceted relationships in data and enhance distillation. Furthermore, a noise- and background-reduction mask (NBRM) is employed to filter noise from transferred knowledge, promoting focus on discriminative features. During training, the student utilizes various sources of supervision, including dynamically improving the teacher's predictions, ground truths, and hash code matching. This assists the student in closely replicating the teacher's abilities despite using fewer parameters. Experimental evaluation on four benchmark datasets - CIFAR-10, CIFAR-100, NUS-WIDE, and ImageNet - demonstrates that OKD outperforms existing hashing methods. OKD achieves 92.98 %, 88.72 %, and 75.88 % mean average precision on CIFAR-10, NUS-WIDE, and ImageNet datasets, respectively, with up to 1.83 %, 1.69 %, and 0.80 % higher accuracy than the previous best methods, across different hash code lengths. By matching teacher ability using distilled knowledge, OKD addresses the barriers that prevent powerful models from being deployed on resource-constrained mobile/embedded platforms.