Evaluation of the epidemic intelligence from open sources (EIOS) system for the early detection of outbreaks and health emergencies in the African region.

Abstract

Introduction: Public health today is challenged by a wide array of hazards that threaten humans, often resulting in high rates of morbidity and mortality when they strike. These events should be detected and responded to as early as possible to save lives and minimize their impact. The Epidemic Intelligence from Open Sources (EIOS) system leverages natural language processing and machine learning techniques for the early detection of public health events from open-source information using an all-hazards approach. In this study, we quantitatively evaluate the performance of the EIOS system for the early detection of outbreaks and health emergencies in the African region.

Methods: We retrospectively searched the EIOS system to determine if a signal was found on the system for each public health event notified to WHO by the 47 countries in the African region from 2018 to 2023. We computed the proportion of public health event detected by the EIOS system, its sensitivity, harmonic mean, and timeliness. We assessed the association between selected predictors (year of report, hazard type, subregion, source type, and language of source) and early detection of public health events on the EIOS system using a multivariable logistic regression model.

Results: We found a detection proportion of 81.0% and a sensitivity of 47.4%, with a harmonic mean of 59.8%. The proportion of events detected steadily increased over the years and sensitivity increased from a baseline of 44.1% in 2018 to 47.3% in 2023. Signals for more than 80.0% of the public health events notified to WHO in 28 countries were detected on the EIOS system. In 22 countries, signals of at least 50% of the public health events were detected early, that is, before official notification from the National Authorities to WHO. The median time between detection on the EIOS system and notification to WHO was zero days. We found that the type of hazard (infectious and zoonotic), the subregion (West and Central Africa), and the type of source (medical and social media) were associated with early detection.

Conclusions: We conclude that the EIOS system performed well in detecting public health events in the African region early. However, some improvements are needed. We recommend increasing social media and local community radio sources on the EIOS system.