What tool do undergraduate pharmacy students prefer when grading systematic review evidence: AMSTAR-2 or ROBIS?

Introduction

While systematic reviews (SRs) are considered the highest form of evidence in the hierarchy, the quality and standard of reviews varies. Two quality assessment tools have been developed to assess the variation in such standards. This study compared the preference, validity, reliability, and applicability of using A Measurement Tool to Assess Systematic Reviews (AMSTAR-2) and the Risk of Bias in Systematic Reviews (ROBIS) for critically appraising evidence by pharmacy students.

Materials and Methods

Students attended eight lectures on evidence-based medicine. Students independently assessed two SRs using AMSTAR-2 and ROBIS. The agreement between both tools were calculated using Spearman's test while interrater reliability was calculated using Fleiss' κ statistics.

Results

Students reported a preference for the AMSTAR-2 tool due to its clear and distinct rating criteria as well as guidance provided by the tool's developer. In comparison, students found the items on the ROBIS tool difficult to judge as it was subjective. A moderate agreement between both tools on the overall domain ratings was noted (Spearman r_s = 0.60). There was slight agreement in the overall confidence using AMSTAR-2 (κ = 0.05; 95% confidence interval [CI]: 0.01–0.12) and the overall domain in ROBIS (κ = 0.09; 95% CI: 0.01–0.16).

Conclusion

The AMSTAR-2 tool had a low level of concordance in ratings of review among students. However, the AMSTAR-2 tool was preferred by students due to the clear guidance and ease of use.