Surrogate Based Post-HOC Calibration for Distributional Shift

Abstract

This paper focuses on improving the calibration performance of the post-hoc approach for the distributional shift. Taking the popular temperature scaling (TS) as a case in point, the key task is finding a matched temperature for the shifted test set. To address this issue, we pose an insight that temperature is strongly correlated with the shifting intensity by a tiny experiment. Based on the finding, we propose a simple yet effective approach named Surrogate Based Temperature Scaling (SBTS), where the surrogate model is trained to map the relationship between temperature and the shifting intensity. Empirical experimental results of various shift types on the CIFAR-10 and CIFAR-100 demonstrate that SBTS can significantly improve the calibration performance under distributional shift.