Dynamic Updating of Psychosis Prediction Models in Individuals at Ultra-High Risk of Psychosis

Abstract

Background

The performance of psychiatric risk calculators can deteriorate over time due to changes in patient population, referral pathways, and medical advances. Such temporal biases in existing models may lead to suboptimal decisions when translated into clinical practice. Methods are available to correct this bias, but no research has been conducted to investigate their utility in psychiatry.

Methods

We aimed to analyze the performance of model updating methods for predicting psychosis onset by 1 year in 780 individuals at ultra-high risk (UHR) of psychosis from the UHR 1000+ cohort, a longitudinal cohort of UHR individuals recruited to research studies at Orygen, Melbourne, Australia, between 1995 and 2020. Model updating was performed using a yearly adjusted model (recalibration), a continuously updated model (refitting), and a continuous Bayesian updating model (dynamic updating) and compared with a static logistic regression prediction model (original) regarding calibration, discrimination, and clinical net benefit.

Results

The original model was poorly calibrated over the entire validation period. All 3 updating methods improved the predictive performance compared with the original model (recalibration: p = .009; refitting: p = .020; dynamic updating: p = .001). The dynamic updating method demonstrated the best predictive performance (Harrell’s C-index = 0.71; 95% CI, 0.60 to 0.82), calibration slope (slope = 1.12; 95% CI, 0.46 to 1.87), and clinical net benefit over the entire validation period.

Conclusions

Dynamic updating of psychosis prediction models may help to mitigate decreases in performance over time. Therefore, existing psychosis prediction models need to be monitored for temporal biases to mitigate potentially harmful decisions.