Efficient knowledge distillation using a shift window target-aware transformer

Abstract

Target-aware Transformer (TaT) knowledge distillation effectively extracts information from intermediate layers but faces high computational costs for large feature maps. While the non-overlapping Patch-group distillation in TaT reduces complexity, it loses boundary information, affecting accuracy. We propose an improved Shifted Windows Target-aware Transformer (Swin TaT) knowledge distillation method, utilizing a hierarchical shift window strategy to preserve boundary information and balance computational efficiency. Our multi-scale approach optimizes Patch-group distillation with dynamic adjustment, ensuring effective local and global feature transfer. This flexible and efficient design enhances distillation performance, addressing previous limitations. The proposed Swin TaT method demonstrates exceptional performance across various architectures, with ResNet18 as the student network. It achieves 73.03% Top-1 accuracy on ImageNet1K, surpassing the SOTA by 1.06% while reducing parameters to approximately 46% less, and improves mIoU by 2.13% on COCOStuff10k.