Validation and refinement of two interpretable models for coronavirus disease 2019 prognosis prediction

Abstract

Objective: To validate two proposed coronavirus disease 2019 (COVID-19) prognosis models, analyze the characteristics of different models, consider the performance of models in predicting different outcomes, and provide new insights into the development and use of artificial intelligence (AI) predictive models in clinical decision-making for COVID-19 and other diseases. Materials and Methods: We compared two proposed prediction models for COVID-19 prognosis that use a decision tree and logistic regression modeling. We evaluated the effectiveness of different model-building strategies using laboratory tests and/or clinical record data, their sensitivity and robustness to the timings of records used and the presence of missing data, and their predictive performance and capabilities in single-site and multicenter settings. Results: The predictive accuracies of the two models after retraining were improved to 93.2% and 93.9%, compared with that of the models directly used, with accuracies of 84.3% and 87.9%, indicating that the prediction models could not be used directly and require retraining based on actual data. In addition, based on the prediction model, new features obtained by model comparison and literature evidence were transferred to integrate the new models with better performance. Conclusions: Comparing the characteristics and differences of datasets used in model training, effective model verification, and a fusion of models is necessary in improving the performance of AI models.