ACS NSQIP Surgical Risk Calculator Accuracy When Operative Risk is Represented by the Principal CPT® code Versus Many Codes.

Abstract

Objective: To determine whether ACS NSQIP risk calculator (RC) accuracy can be improved by incorporating CPT codes beyond the principal code.

Background: Because of technical limitations, past and current RC algorithms have relied only on the principal CPT code, represented as a logit score, to adjust for procedure-related risk. RC performance was evaluated when using a new machine learning (ML) algorithm capable of incorporating an indeterminate number of high cardinality categorical variables (in this case, multiple CPT codes).

Methods: ACS-NSQIP data from 5,020,713 patients from 2016-2020 were used. Predictive accuracy, for 13 outcomes, was assessed when the RC relied on, in addition to standard predictors, a logit score associated with the principal CPT code (extreme gradient boosting ML, XGB), or up to 21 codes in native categorical form (CatBoost ML, CATB). 80% of cases were used for training and 20% for validation. Discrimination (area under the receiver operator characteristic curve and area under the precision recall curve) and calibration (Hosmer-Lemeshow statistics) were assessed on the entire validation dataset and on a subset of that data that included only patients who had at least 1 CPT code recorded beyond the principal code.

Results: There was no consistent accuracy advantage of CATB over XGB with respect to discrimination. XGB tended to have slightly better calibration than CATB when evaluated on the complete validation dataset but tended to have slightly worse calibration compared to CATB when the validation dataset was limited to the subset of 34.8% of cases where there was at least one code in addition to the principal CPT code. However, there was a subset of patients with 4 or more CPTs (about 8% of all patients) where CATB provided meaningfully more accurate estimates than XGB.

Conclusions: While the current RC, relying on XGB and the principal CPT code, remains a viable approach to routine surgical risk assessment, an advanced version of the RC, based on the CATB algorithm and accommodating multiple CPT codes, may provide more accurate estimates.