A triple semantic-aware knowledge distillation network for industrial defect detection

Abstract

Knowledge distillation (KD) is a powerful model compression technique that aims to transfer knowledge from heavy teacher networks to compact student networks via distillation. However, effectively transferring semantic knowledge in industrial settings poses significant challenges. On one hand, the appearance of defects (e.g., size and shape) may vary considerably due to the influence of the industrial site, which potentially weakens the semantic associations between class-specific features. On the other hand, agnostic background interference (e.g., spike anomalies and low light) may foster semantic ambiguity of class-specific features. As such, the weakened semantic associations and fostered semantic ambiguities hinder the efficacy and adequacy of knowledge transfer in KD. To mitigate these limitations, we propose a triple semantic-aware knowledge distillation (TSKD) network for industrial defect detection. TSKD contains three refinements, i.e., dual-relation distillation (DRD), decoupled expert distillation (DED), and cross-response distillation (CRD). Specifically, DRD employs graph reasoning networks to strengthen semantic associations at both the instance and pixel levels, DED enhances semantic explicitness by decoupling foreground and background features while injecting expert priors, and CRD further captures task-specific semantic response knowledge. By integrating these components, TSKD can effectively perceive triple semantic knowledge of relations, features, and responses, ensuring more robust and comprehensive knowledge transfer. Experimental evaluations on two challenging industrial datasets show that TSKD can significantly improve detector performance (MFL-DET: 98.9% mAP; NEU-DET: 81.0% mAP) and compress computation (MFL-DET: 19.7M Params and 105 FPS; NEU-DET: 19.7M Params and 116 FPS).