Development and Internal-External Validation of a Post-Operative Mortality Risk Calculator for Pediatric Surgical Patients in Low- and Middle- Income Countries Using Machine Learning

Abstract

Background

The purpose of this study was to develop and validate a mortality risk algorithm for pediatric surgery patients treated at KidsOR sites in 14 low- and middle-income countries.

Methods

A SuperLearner machine learning algorithm was trained to predict post-operative mortality by hospital discharge using the retrospectively and prospectively collected KidsOR database including patients treated at 20 KidsOR sites from June 2018 to June 2023. Algorithm performance was evaluated by internal-external cross-validated AUC and calibration.

Findings

Of 23,905 eligible patients, 21,703 with discharge status recorded were included in the analysis, representing a post-operative mortality rate of 3.1% (671 mortality events). The candidate algorithm with the best cross-validated performance was an extreme gradient boosting model. The cross-validated AUC was 0.945 (95% CI 0.936 to 0.954) and cross-validated calibration slope and intercept were 1.01 (95% CI 0.96 to 1.06) and 0.05 (95% CI -0.10 to 0.21). For Super Learner models trained on all but one site and evaluated in the holdout site for sites with at least 25 mortality events, overall external validation AUC was 0.864 (95% CI 0.846 to 0.882) with calibration slope and intercept of 1.03 (95% CI 0.97 to 1.09) and 1.18 (95% CI 0.98 to 1.39).

Interpretation

The KidsOR post-operative mortality risk algorithm had outstanding cross-validated discrimination and strong cross-validated calibration. Across all external validation sites, discrimination of Super Learner models trained on the remaining sites was excellent, though re-calibration may be necessary prior to use at new sites. This model has the potential to inform clinical practice and guide resource allocation at KidsOR sites world-wide.

Type of Study and Level of Evidence

Observational Study, Level III.