Combining Knowledge Distillation and Transfer Learning for Sensor Fusion in Visible and Thermal Camera-based Person Classification

Visible and thermal camera-based sensor fusion has shown to address the limitations and enhance the robustness of visible camera-based person classification. In this paper, we propose to further enhance the classification accuracy of visible-thermal person classification using transfer learning, knowledge distillation, and the vision transformer. In our work, the visible-thermal person classifier is implemented using the vision transformer. The proposed classifier is trained using the transfer learning and knowledge distillation techniques. To train the proposed classifier, visible and thermal teacher models are implemented using the vision transformers. The multimodal classifier learns from the two teachers using a novel loss function which incorporates the knowledge distillation. The proposed method is validated on the public Speaking Faces dataset. A comparative analysis with baseline algorithms and an ablation study is performed. The results show that the proposed framework reports an enhanced classification accuracy.