Real-World evaluation of an AI triaging system for chest X-rays: A prospective clinical study

Abstract

Chest X-rays (CXRs) are crucial for diagnosing and managing lung conditions. While CXR is a common and cost-effective diagnostic tool, interpreting the high volume of CXRs is challenging due to workforce limitations. Artificial intelligence (AI) offers promise in enhancing efficiency and accuracy. However, real-world applicability and generalizability across diverse patient cohorts remain areas of concerns. In our study, the LUNIT INSIGHT CXR Triage software was evaluated in a diverse patient cohort. Forty-three radiologists, blinded to AI results, assessed CXRs categorized into normal, non-urgent, and urgent using a 3-tier classification system. Performance metrics and turnaround times were analyzed.

The AI system demonstrated sensitivity of 89% for normal CXRs, specificity of 93%, PPV of 83%, and NPV of 95%, with an F1 score of 0.86 and an AUC of 0.91. For non-urgent CXRs, sensitivity and specificity were 93% and 91%, with PPV and NPV at 94% and 89%, respectively, and an F1 score of 0.94 and an AUC of 0.92. In the urgent category, sensitivity was 82%, specificity 99%, PPV 90%, and NPV 98%. Subgroup analysis revealed consistently high accuracy across various age groups (Young, Adult, Senior), genders, and ethnicities (Chinese, Malay, Indian, Others), with sensitivity, specificity, and AUC consistently above 84%. The AI system also significantly reduced turnaround times across all subgroups, indicating its robust performance and generalizability in diverse healthcare settings.