Can using the Cochrane RCT classifier in EPPI-Reviewer help speed up study selection in qualitative evidence syntheses? A retrospective evaluation

Abstract

Introduction

Using machine learning functions, such as study design classifiers, to automatically identify studies that do not meet the inclusion criteria, is one way to speed up the systematic review screening process. As a qualitative study design classifier is yet to be developed, using the Cochrane randomized controlled trial (RCT) classifier in reverse is one possible way to speed up the identification of primary qualitative studies during screening. The objective of this study was to evaluate whether the Cochrane RCT classifier can be used to speed up the study selection process for qualitative evidence synthesis (QES).

Methods

We performed a retrospective evaluation where we first identified QES. We then extracted the bibliographic information of the included primary qualitative studies in each QES, and uploaded the references into our data management tool, EPPI-Reviewer. We then ran the Cochrane RCT classifier on each group of included studies for each QES.

Results

Eighty-two QES with 2828 unique primary studies were included in the analysis. 56% of the primary studies were classified as unlikely to be an RCT and 40% as being 0–9% likely to be an RCT. 4% were classified as being 10% or more likely to be an RCT. Of these, only 1.7% were classified as being 50% or more likely to be an RCT.

Conclusions

The Cochrane RCT classifier could be a useful tool to identify primary studies with qualitative study designs to speed up study selection in a QES. However, it is possible that mixed methods studies or qualitative studies conducted as part of a clinical trial may be missed. Further evaluations using the Cochrane RCT classifier on all the references retrieved from the complete literature search is needed to investigate time- and resource savings.