Real-World Performance of Pneumothorax-Detecting Artificial Intelligence Algorithm and its Impact on Radiologist Reporting Times.

Abstract

Rationale and objectives: Artificial intelligence (AI) algorithms in radiology capable of detecting urgent findings have gained significant traction in recent years, but the impact of these algorithms on real-world clinical practice remains unclear with need for scientific investigation. Our study investigates the diagnostic accuracy and impact on radiologist report turnaround times of an FDA-approved AI tool for pneumothorax (PTx) detection on inpatient chest X-rays (CXR) in our institution's radiology practice at a large academic medical center.

Materials and methods: This retrospective study included 27,397 frontal, single-view CXRs of adult inpatients collected consecutively between August 2020 and April 2021 following deployment of an AI-based PTx detection and picture archiving and communication system (PACS) alert system. 12,728 CXRs were acquired within the AI-integrated system while 14,669 CXRs were acquired outside of the system. Receiver operator characteristic (ROC) analysis was conducted with final radiology reports as the reference standard to evaluate diagnostic accuracy of the AI algorithm in detection of PTx. Wilcoxon rank sum tests were conducted to evaluate the effect of the AI-integrated alert system on radiologist reporting times.

Results: Area under ROC curve (AUC) for the AI tool was.78 with sensitivity of .60 and specificity of .97. When selecting for moderate/large PTx, AUC, sensitivity and specificity increased to .93, .89 and .96, respectively. Median reporting time in CXRs with radiologist-confirmed PTx was reduced by 46% in those with AI integration as compared to those without AI integration (100 vs. 186 min, p < .001).

Conclusion: Real-world deployment of an AI-integrated system capable of detecting PTx and generating alerts within PACS achieved a strong AUC for clinically actionable PTx (i.e., moderate- or large-sized) while substantially reducing median radiologist reporting times, enabling swifter clinical response to a critical but treatable condition.