International multi-institutional external validation of preoperative risk scores for 30-day in-hospital mortality in paediatric patients

Abstract

Background

Risk prediction scores are used to guide clinical decision-making. Our primary objective was to externally validate two patient-specific risk scores for 30-day in-hospital mortality using the Multicenter Perioperative Outcomes Group (MPOG) registry: the Pediatric Risk Assessment (PRAm) score and the intrinsic surgical risk score. The secondary objective was to recalibrate these scores.

Methods

Data from 56 US and Dutch hospitals with paediatric caseloads were included. The primary outcome was 30-day mortality. To assess model discrimination, the area under the receiver operating characteristic curve (AUROC) and area under the precision–recall curve (AUC-PR) were calculated. Model calibration was assessed by plotting the observed and predicted probabilities. Decision analytic curves were fit.

Results

The 30-day mortality was 0.14% (822/606 488). The AUROC for the PRAm upon external validation was 0.856 (95% confidence interval 0.844–0.869), and the AUC-PR was 0.008. Upon recalibration, the AUROC was 0.873 (0.861–0.886), and the AUC-PR was 0.031. The AUROC for the external validation of the intrinsic surgical risk score was 0.925 (0.914–0.936) and AUC-PR was 0.085. Upon recalibration, the AUROC was 0.925 (0.915–0.936), and the AUC-PR was 0.094. Calibration metrics for both scores were favourable because of the large cluster of cases with low probabilities of mortality. Decision curve analyses showed limited benefit to using either score.

Conclusions

The intrinsic surgical risk score performed better than the PRAm, but both resulted in large numbers of false positives. Both scores exhibited decreased performance compared with the original studies. ASA physical status scores in sicker patients drove the superior performance of the intrinsic surgical risk score, suggesting the use of a risk score does not improve prediction.