Evaluating the Performance of a Commercially Available Artificial Intelligence Algorithm for Automated Detection of Pulmonary Embolism on Contrast-Enhanced Computed Tomography and Computed Tomography Pulmonary Angiography in Patients With Coronavirus Disease 2019

Abstract

Objective

To investigate the performance of a commercially available artificial intelligence (AI) algorithm for the detection of pulmonary embolism (PE) on contrast-enhanced computed tomography (CT) scans in patients hospitalized for coronavirus disease 2019 (COVID-19).

Patients and Methods

Retrospective analysis was performed of all contrast-enhanced chest CT scans of patients admitted for COVID-19 between March 1, 2020 and December 31, 2021. Based on the original radiology reports, all PE-positive examinations were included (n=527). Using a reversed-flow single-gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative examinations (n=977) was included. Pulmonary parenchymal disease severity was assessed for all the included studies using a semiquantitative system, the total severity score. All included CT scans were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by 3 blinded radiologists, who rendered a final determination of indeterminate, positive, or negative.

Results

A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and were excluded. The sensitivity and specificity of AI were 93.2% (95% CI, 90.6%-95.2%) and 99.6% (95% CI, 98.9%-99.9%), respectively. The accuracy of AI for all total severity score groups (mild, moderate, and severe) was high (98.4%, 96.7%, and 97.2%, respectively). Artificial intelligence was more accurate in PE detection on CT pulmonary angiography scans than on contrast-enhanced CT scans (P<.001), with an optimal Hounsfield unit of 362 (P=.048).

Conclusion

The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast-enhanced CT scans in patients with COVID-19 regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.