SHAP-assisted EE-LightGBM model for explainable fault diagnosis in practical optical networks

Abstract

Reliable fault diagnosis is crucial for ensuring the stable operation of optical networks. Recently, data-driven techniques have demonstrated significant advantages in fault diagnosis due to their outstanding data-processing capabilities and adaptive learning abilities. However, as equipment faults in practical optical networks are rare events, the data collected often faces severe data imbalance issues, greatly limiting the accuracy of traditional data-driven models. To address this challenge, a SHAP-assisted EE-LightGBM scheme is proposed for explainable fault diagnosis in practical optical networks. The EE-LightGBM model integrates undersampling strategies at the data level and hybrid ensemble strategies at the model level, enabling the full utilization of fewer fault samples and effectively alleviating the impact of data imbalance on model training. Furthermore, the SHAP method is used to explain the EE-LightGBM model. This method quantifies the contributions of input features to the model’s decision outputs, facilitating a deeper understanding of the mechanisms underlying faults in the equipment and improving the model’s explainability. Through SHAP analysis, we can determine key features highly correlated with equipment faults, thereby inferring the causes of equipment faults. Evaluation using data from backbone network equipment managed by operators shows excellent detection performance of the EE-LightGBM model at a data imbalance rate of 5.61%, with accuracy and F1 scores of 0.9968 and 0.9711, and false negative and false positive rates of 0.0033 and 0.0032, respectively. Moreover, the cause identification results are consistent with diagnostic expertise. We also explore the impact of data imbalance rates on the detection performance of the EE-LightGBM model. The model’s low false negative rate under data imbalance further demonstrates its effectiveness in practical optical network fault diagnosis.