Knowledge Probabilization in Ensemble Distillation: Improving Accuracy and Uncertainty Quantification for Object Detectors

Abstract

Ensemble object detectors have demonstrated remarkable effectiveness in enhancing prediction accuracy and uncertainty quantification. However, their widespread adoption is hindered by significant computational and storage demands, limiting their feasibility in resource-constrained settings. To overcome this, researchers have focused on distilling the knowledge from ensemble object detectors into a single model. In this article, we introduce probabilization based ensemble distillation (ProbED), an innovative ensemble distillation framework that consolidates knowledge from multiple object detectors into a single, resource-efficient model. Unlike traditional ensemble distillation methods that average the outputs of subteachers, ProbED captures comprehensive outcome distributions from all subteachers, providing a more nuanced approach to knowledge transfer. ProbED employs knowledge probabilization to achieve a sophisticated and refined aggregation of teacher knowledge, including feature knowledge, semantic knowledge, and localization knowledge, resulting in dual improvements in prediction accuracy and uncertainty quantification for the student model. In particular, ProED's novel knowledge probabilization-based approach to aggregating teacher knowledge is inspired by our empirical observations, which demonstrate that knowledge probabilization excels in effectively representing uncertainty, improving prediction, and facilitating robust knowledge transfer. Furthermore, we introduce a random smoothing perturbation technique to modify inputs within ProbED, further enhancing the distillation process. Extensive experiments highlight ProbED's ability to significantly enhance the prediction accuracy and uncertainty quantification of various object detectors, demonstrating its superior performance compared to other state-of-the-art techniques.