Cross-Modal Knowledge Distillation with Dropout-Based Confidence

Abstract

In cross-modal distillation, e.g., from text-based inference modules to spoken language understanding module, it is difficult to determine the teacher's influence due to the different nature of both modalities that bring the heterogeneity in the aspect of uncertainty. Though error rate or entropy-based schemes have been suggested to cope with the heuristics of time-based scheduling, the confidence of the teacher inference has not been necessarily taken into deciding the teacher's influence. In this paper, we propose a dropout-based confidence that decides the teacher's confidence and to-student influence of the loss. On the widely used spoken language understanding dataset, Fluent Speech Command, we show that our weight decision scheme enhances performance in combination with the conventional scheduling strategies, displaying a maximum 20% relative error reduction concerning the model with no distillation.