Label-free evaluation for performance of fault diagnosis model on unknown distribution dataset

Abstract

Real-time data may undergo distribution drift due to changes in operating conditions and other factors, which can affect the classification accuracy of online fault diagnosis models and potentially lead to serious consequences. Therefore, understanding the classification accuracy of the model on real-time data holds substantial significance. However, the absence of labels in real-time data presents a challenge for evaluating classification accuracy. Furthermore, the real-time nature of fault diagnosis necessitates a swift and straightforward evaluation method. For the above reasons, this paper proposes a method for evaluating the classification accuracy of a model on real-time data, which is done in the absence of labels for the real-time data. The proposed label-free evaluation method transforms the model’s output into a scalar that measures the degree of matching between the classification probabilities, termed the average free energy. It then establishes a mapping between the average free energy and the classification accuracy using an auxiliary dataset. Finally, it predicts the model’s classification accuracy on the real-time data through this mapping and the average free energy of the real-time data. The proposed method is experimentally evaluated on public datasets, demonstrating its superiority in various aspects.