Systematic literature review and meta-analysis for real-world versus clinical validation performance of artificial intelligence applications indicated for ICH and LVO detection

Abstract

Purpose

We sought to compare the performance of AI applications in real-world studies to validation study data used to gain regulatory approval.

Methods

We searched PubMed, EBSCO, and EMBASE for publications from 2018 to 2023. We included articles that evaluated the sensitivity and specificity of ICH and LVO detection applications in real-world populations. We performed a quality and applicability assessment using QUADAS-2. We used a bivariate or two univariate meta-analyses, where appropriate, to calculate summary point estimates for sensitivity and specificity.

Results

Eighteen articles met the criteria of the systematic literature review. The included articles evaluated five applications indicated for ICH or LVO triage. Three of the five applications yielded adequate studies to be included in the meta-analysis. For most applications, we did not observe any systematic differences in sensitivity and specificity results between the point estimates from the meta-analysis and the respective 510k studies. For VIZ LVO and RAPID LVO, the 95 % CI for real-world sensitivity sat within the 95 % CI from their respective validation study. For BriefCase ICH, the 95 % CI for real-world sensitivity sat below the 95 % CI of the respective validation study. Additionally, the 95 % CI for real-world specificity for all three of the applications sat within the 95 % CI of their respective validation studies. Data from the individual real-world studies for RAPID ICH and CINA LVO followed a similar trend.

Conclusion

The performance of applications in real-world settings was non-inferior to the performance observed in validation studies used to obtain 510k clearance.