Implementation of a dynamic model updating pipeline provides a systematic process for maintaining performance of prediction models

Abstract

Objectives

We describe the steps for implementing a dynamic updating pipeline for clinical prediction models and illustrate the proposed methods in an application of 5-year survival prediction in cystic fibrosis.

Study Design and Setting

Dynamic model updating refers to the process of repeated updating of a clinical prediction model with new information to counter performance degradation. We describe 2 types of updating pipeline: “proactive updating” where candidate model updates are tested any time new data are available, and “reactive updating” where updates are only made when performance of the current model declines or the model structure changes. Methods for selecting the best candidate updating model are based on measures of predictive performance under the 2 pipelines. The methods are illustrated in our motivating example of a 5-year survival prediction model in cystic fibrosis. Over a dynamic updating period of 10 years, we report the updating decisions made and the performance of the prediction models selected under each pipeline.

Results

Both the proactive and reactive updating pipelines produced survival prediction models that overall had better performance in terms of calibration and discrimination than a model that was not updated. Further, use of the dynamic updating pipelines ensured that the prediction model’s performance was consistently and frequently reviewed in new data.

Conclusion

Implementing a dynamic updating pipeline will help guard against model performance degradation while ensuring that the updating process is principled and data-driven.