Assessing the Robustness of a Machine Learning Model for Predicting Asthma Hospital Encounters During the COVID-19 Pandemic

Abstract

Rationale: Asthma imposes a significant burden on healthcare. To better allocate limited preventive care management resources and improve patient outcomes, we recently used machine learning to build the world’s most accurate model to predict asthma hospital encounters (emergency department visits or inpatient stays for asthma) in the subsequent 12 months. Under pre-pandemic conditions, the model reached an area under the receiver operating characteristic curve (AUC) of >0.9. Performance of the model under COVID-19 conditions had not been previously tested. Objective: To assess the robustness of our predictive model during the COVID-19 pandemic. Methods: The patient cohort included all 55,435 adult asthmatic patients who visited the University of Washington Medicine facilities between 2011-2020. For each year t (2011≤ t ≤2020) , the corresponding effective data were of the patients who had asthma in t , included 71 features computed on these patients’ data during 2011-t , and contained these patients’ outcomes in t +1 as the prediction targets. For each outcome year y (2019 ≤ y ≤ 2021), the model was trained on the 2011-( y − 2) effective data and then tested on the y − 1 effective data. Results: The model yielded an AUC of 0.929, 0.892, and 0.904 for the outcome years 2019 (pre-pandemic), 2020 (peri-pandemic