Systematic reviewers' perspectives on replication of systematic reviews: A survey

Cochrane Evidence Synthesis and Methods Pub Date : 2023-04-10 DOI:10.1002/cesm.12009

Phi-Yen Nguyen, Joanne E. McKenzie, Daniel G. Hamilton, David Moher, Peter Tugwell, Fiona M. Fidler, Neal R. Haddaway, Julian P. T. Higgins, Raju Kanukula, Sathya Karunananthan, Lara J. Maxwell, Steve McDonald, Shinichi Nakagawa, David Nunan, Vivian A. Welch, Matthew J. Page

{"title":"Systematic reviewers' perspectives on replication of systematic reviews: A survey","authors":"Phi-Yen Nguyen, Joanne E. McKenzie, Daniel G. Hamilton, David Moher, Peter Tugwell, Fiona M. Fidler, Neal R. Haddaway, Julian P. T. Higgins, Raju Kanukula, Sathya Karunananthan, Lara J. Maxwell, Steve McDonald, Shinichi Nakagawa, David Nunan, Vivian A. Welch, Matthew J. Page","doi":"10.1002/cesm.12009","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Replication is essential to the scientific method. It is unclear what systematic reviewers think about the replication of systematic reviews (SRs). Therefore, we aimed to explore systematic reviewers' perspectives on (a) the definition and importance of SR replication; (b) incentives and barriers to conducting SR replication; and (c) a checklist to guide when to replicate an SR.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>We searched PubMed for SRs published from January to April 2021, from which we randomly allocated 50% to this survey and 50% to another survey on data sharing in SRs. We sent an electronic survey to authors of these SRs (<i>n</i> = 4669) using Qualtrics. Quantitative responses were summarized using frequency analysis. Free-text answers were coded using an inductive approach.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>The response rate was 9% (<i>n</i> = 409). Most participants considered “replication of SRs” as redoing an SR (68%) or reanalyzing originally collected data (61%), using the same or similar methods. Participants also considered updating an SR, either one's own (42%) or others (43%), equivalent to replication. Most participants agreed that replication of SRs is important (89%). Although 54% of participants reported having conducted a replication of a SR, only 22% have published a replication within 5 years. Those who published a replication (<i>n</i> = 89) often found their replication supported (47%) or expanded the generalizability of the original review (51%). The most common perceived barriers to replicating SRs were difficulty publishing (75%), less prestige (65%), fewer citations (56%), and less impact on career advancement (55%) compared to conducting an original SR. A checklist to assess the need for replication was deemed useful (79%) and easy to apply in practice (69%) by participants.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>Reviewers have various perceptions of what constitutes a replication of SRs. Reviewers see replication as important and valuable but perceive several barriers to conducting replications. Institutional support should be better communicated to reviewers to address these perceptions.</p>\n </section>\n </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"1 2","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2023-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.12009","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Cochrane Evidence Synthesis and Methods","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/cesm.12009","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 2

Abstract

Background

Replication is essential to the scientific method. It is unclear what systematic reviewers think about the replication of systematic reviews (SRs). Therefore, we aimed to explore systematic reviewers' perspectives on (a) the definition and importance of SR replication; (b) incentives and barriers to conducting SR replication; and (c) a checklist to guide when to replicate an SR.

Methods

We searched PubMed for SRs published from January to April 2021, from which we randomly allocated 50% to this survey and 50% to another survey on data sharing in SRs. We sent an electronic survey to authors of these SRs (n = 4669) using Qualtrics. Quantitative responses were summarized using frequency analysis. Free-text answers were coded using an inductive approach.

Results

The response rate was 9% (n = 409). Most participants considered “replication of SRs” as redoing an SR (68%) or reanalyzing originally collected data (61%), using the same or similar methods. Participants also considered updating an SR, either one's own (42%) or others (43%), equivalent to replication. Most participants agreed that replication of SRs is important (89%). Although 54% of participants reported having conducted a replication of a SR, only 22% have published a replication within 5 years. Those who published a replication (n = 89) often found their replication supported (47%) or expanded the generalizability of the original review (51%). The most common perceived barriers to replicating SRs were difficulty publishing (75%), less prestige (65%), fewer citations (56%), and less impact on career advancement (55%) compared to conducting an original SR. A checklist to assess the need for replication was deemed useful (79%) and easy to apply in practice (69%) by participants.

Conclusion

Reviewers have various perceptions of what constitutes a replication of SRs. Reviewers see replication as important and valuable but perceive several barriers to conducting replications. Institutional support should be better communicated to reviewers to address these perceptions.

Abstract Image

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

系统评价者对系统评价复制的看法：一项调查

背景复制对科学方法至关重要。目前尚不清楚系统评审员对系统评审的复制有何看法。因此，我们旨在探讨系统评审者对（a）SR复制的定义和重要性的看法；（b）开展SR复制的激励措施和障碍；和（c）指导何时复制SR的检查表。方法我们在PubMed中搜索2021年1月至4月发布的SR，从中我们随机分配50%用于本次调查，50%用于另一项关于SR数据共享的调查。我们向这些SR（n = 4669）。使用频率分析对定量反应进行总结。自由文本答案采用归纳法进行编码。结果有效率为9%（n = 409）。大多数参与者认为“SR的复制”是使用相同或类似的方法重做SR（68%）或重新分析原始收集的数据（61%）。参与者还考虑更新SR，无论是自己的（42%）还是其他人的（43%），相当于复制。大多数参与者都认为复制SR很重要（89%）。尽管54%的参与者报告进行了SR的复制，但只有22%的参与者在5年内发表了复制。发布复制（n = 89）经常发现他们的复制得到支持（47%）或扩大了原始综述的可推广性（51%）。与进行原始SR相比，复制SR最常见的障碍是出版困难（75%）、声望较低（65%）、引用次数较少（56%）以及对职业发展的影响较小（55%）。评估复制需求的检查表被参与者认为是有用的（79%），并且易于在实践中应用（69%）。结论评论者对什么是SRs的复制有不同的看法。评审员认为复制是重要和有价值的，但认为进行复制存在一些障碍。应更好地向审查人员传达机构支持，以解决这些看法。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Cochrane Evidence Synthesis and Methods

自引率

0.00%

发文量