Frequency of use and adequacy of Cochrane risk of bias tool 2 in non-Cochrane systematic reviews published in 2020: Meta-research study

Abstract

Risk of bias (RoB) assessment is essential to the systematic review methodology. The new version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019 to address limitations identified since the first version of the tool was published in 2008 and to increase the reliability of assessments. This study analyzed the frequency of usage of the RoB 2 and the adequacy of reporting the RoB 2 assessments in non-Cochrane reviews published in 2020. This meta-research study included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that used the RoB 2 tool, we analyzed the reporting of the RoB 2 assessment. Among 3880 included reviews, the Cochrane RoB 1 tool was the most frequently used (N = 2228; 57.4%), followed by the Cochrane RoB 2 tool (N = 267; 6.9%). From 267 reviews that reported using the RoB 2 tool, 213 (79.8%) actually used it. In 26 (12.2%) reviews, erroneous statements were used to indicate the RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment with a detailed table of answers to all signaling questions. The judgment of risk of bias by the RoB 2 tool was not justified by a comment in 158 (74.2%) reviews. Only in 33 (14.5%) of reviews the judgment in all domains was justified in the accompanying comment. In most reviews (81.7%), the RoB was inadequately assessed at the study level. In conclusion, the majority of non-Cochrane reviews published in 2020 still used the Cochrane RoB 1 tool. Many reviews used the RoB 2 tool inadequately. Further studies about the uptake and the use of the RoB 2 tool are needed.