Development and validation of an interpretable machine learning model for predicting in-hospital mortality for ischemic stroke patients in ICU

Abstract

Background

Timely and accurate outcome prediction is essential for clinical decision-making for ischemic stroke patients in the intensive care unit (ICU). However, the interpretation and translation of predictive models into clinical applications are equally crucial. This study aims to develop an interpretable machine learning (IML) model that effectively predicts in-hospital mortality for ischemic stroke patients.

Methods

In this study, an IML model was developed and validated using multicenter cohorts of 3225 ischemic stroke patients admitted to the ICU. Nine machine learning (ML) models, including logistic regression (LR), K-nearest neighbors (KNN), naive Bayes (NB), decision tree (DT), support vector machine (SVM), random forest (RF), XGBoost, LightGBM, and artificial neural network (ANN), were developed to predict in-hospital mortality using data from the MIMIC-IV and externally validated in Shanghai Changhai Hospital. Feature selection was conducted using three algorithms. Model’s performance was assessed using area under the receiver operating characteristic (AUROC), accuracy, sensitivity, specificity and F1 score. Calibration curve and Brier score were used to evaluate the degree of calibration of the model, and decision curve analysis were generated to assess the net clinical benefit. Additionally, the SHapley Additive exPlanations (SHAP) method was employed to evaluate the risk of in-hospital mortality among ischemic stroke patients admitted to the ICU.

Results

Mechanical ventilation, age, statins, white blood cell, blood urea nitrogen, hematocrit, warfarin, bicarbonate and systolic blood pressure were selected as the nine most influential variables. The RF model demonstrated the most robust predictive performance, achieving AUROC values of 0.908 and 0.858 in the testing set and external validation set, respectively. Calibration curves also revealed a high consistency between observations and predictions. Decision curve analysis showed that the model had the greatest net benefit rate when the prediction probability threshold is 0.10 ∼ 0.80. SHAP was employed to interpret the RF model. In addition, we have developed an online prediction calculator for ischemic stroke patients.

Conclusion

This study develops a machine learning–based calculator to predict the probability of in-hospital mortality among patients with ischemic stroke in ICU. The calculator has the potential to guide clinical decision-making and improve the care of patients with ischemic stroke by identifying patients at a higher risk of in-hospital mortality.