External validation and comparison of four prediction scores for severe maternal morbidity

Background

Severe maternal morbidity (SMM) is increasing in the United States. Several tools and scores exist to stratify an individual's risk of SMM.

Objective

We sought to examine and compare the validity of four scoring systems for predicting SMM.

Study Design

This was a retrospective cohort study of all individuals in the Consortium on Safe Labor dataset, which was conducted from 2002 to 2008. Individuals were excluded if they had missing information on risk factors. SMM was defined based on the Centers for Disease Control and Prevention excluding blood transfusion. Blood transfusion was excluded due to concerns regarding the specificity of International Classification of Diseases codes for this indicator and its variable clinical significance. Risk scores were calculated for each participant using the Assessment of Perinatal Excellence (APEX), California Maternal Quality Care Collaborative (CMQCC), Obstetric Comorbidity Index (OB-CMI), and modified OB-CMI. We calculated the probability of SMM according to the risk scores. The discriminative performance of the prediction score was examined by the areas under receiver operating characteristic curves and their 95% confidence intervals (95% CI). The area under the curve for each score was compared using the bootstrap resampling. Calibration plots were developed for each score to examine the goodness-of-fit. The concordance probability method was used to define an optimal cutoff point for the best-performing score.

Results

Of 153, 463 individuals, 1115 (0.7%) had SMM. The CMQCC scoring system had a significantly higher area under the curve (95% CI) (0.78 [0.77–0.80]) compared to the APEX scoring system, OB-CMI, and modified OB-CMI scoring systems (0.75 [0.73–0.76], 0.67 [0.65–0.68], 0.66 [0.70–0.73]; P<.001). Calibration plots showed excellent concordance between the predicted and actual SMM for the APEX scoring system and OB-CMI (both Hosmer–Lemeshow test P values=1.00, suggesting goodness-of-fit).

Conclusion

This study validated four risk-scoring systems to predict SMM. Both CMQCC and APEX scoring systems had good discrimination to predict SMM. The APEX score and the OB-CMI had goodness-of-fit. At ideal calculated cut-off points, the APEX score had the highest sensitivity of the four scores at 71%, indicating that better scoring systems are still needed for predicting SMM.