Methodological evaluation of systematic reviews based on the use of artificial intelligence systems in chest radiography

Abstract

Introduction

In recent years, systems that use artificial intelligence (AI) in medical imaging have been developed, such as the interpretation of chest X-ray to rule out pathology. This has produced an increase in systematic reviews (SR) published on this topic. This article aims to evaluate the methodological quality of SRs that use AI for the diagnosis of thoracic pathology by simple chest X-ray.

Material and methods

SRs evaluating the use of AI systems for the automatic reading of chest X-ray were selected. Searches were conducted (from inception to May 2022): PubMed, EMBASE, and the Cochrane Database of Systematic Reviews. Two investigators selected the reviews. From each SR, general, methodological and transparency characteristics were extracted. The PRISMA statement for diagnostic tests (PRISMA-DTA) and AMSTAR-2 were used. A narrative synthesis of the evidence was performed. Protocol registry: Open Science Framework: https://osf.io/4b6u2/.

Results

After applying the inclusion and exclusion criteria, 7 SRs were selected (mean of 36 included studies per review). All the included SRs evaluated “deep learning” systems in which chest X-ray was used for the diagnosis of infectious diseases. Only 2 (29%) SRs indicated the existence of a review protocol. None of the SRs specified the design of the included studies or provided a list of excluded studies with their justification. Six (86%) SRs mentioned the use of PRISMA or one of its extensions. The risk of bias assessment was performed in 4 (57%) SRs. One (14%) SR included studies with some validation of AI techniques. Five (71%) SRs presented results in favour of the diagnostic capacity of the intervention. All SRs were rated "critically low" following AMSTAR-2 criteria.

Conclusions

The methodological quality of SRs that use AI systems in chest radiography can be improved. The lack of compliance in some items of the tools used means that the SRs published in this field must be interpreted with caution.