Current intelligent fault diagnosis studies focus on improving model accuracy. While accuracy is crucial, an exclusive emphasis on this metric can leave users oblivious to potentially untrustworthy decisions made by the model. This underscores the importance of confidence estimation and brings the model miscalibration problem to the forefront, i.e., the softmax probability, which is supposed to indicate the likelihood of the predicted label being correct but fails to reflect the true probability accurately. Addressing this issue is imperative for several reasons. Firstly, a well-calibrated model can provide users with an assessment of the risk associated with prediction failures, thereby withholding decision-making when the confidence is low and mitigating the risk of erroneous outputs. Especially in situations involving out-of-distribution (OOD) and distribution-shifted inputs, where the risk of model failure increases, the calibration property becomes even more critical. Secondly, well-calibrated confidence estimates can enhance users’ trust in today’s many black-box models. However, there have been limited fault diagnosis studies that specifically explore model calibration. The effectiveness of existing calibration methods in handling OOD and distribution-shifted inputs also remains unclear. Therefore, this paper evaluates multiple calibration methods and discusses their advantages and limitations, providing insights for subsequent studies. The results suggest that a deep ensemble method, which derives predictive expectations using multiple models with significantly different structures or parameters, has the potential to be the best calibration method. Code used in this paper is available at https://github.com/xiaoyiming1999/Calibration_for_RMFD.
扫码关注我们
求助内容:
应助结果提醒方式:
