Areti Angeliki Veroniki, Ivan Florez, Brian Hutton, Sharon E. Straus, Andrea C. Tricco
Recently, Ades and colleagues discussed the controversies and advancements in network meta-analysis (NMA) over the past two decades, discussing its reliability, assumptions, novel approaches, and provided some useful recommendations for the conduction of NMAs. The present discussion paper builds on the insights by Ades and colleagues, providing a roadmap for NMA applications, advancements in software and tools, and approaches designed to facilitate the assessment and interpretation of NMA findings. It also discusses the impact of NMA across disciplines, particularly for policymakers and guideline developers. Despite 20 years of NMA history, challenges remain in understanding and assessing assumptions, communicating and interpreting findings, and applying common approaches like network meta-regression and NMA involving non-randomized studies in readily available software. NMA has proven particularly valuable in clinical decision-making, which highlights the need for additional training and interdisciplinary collaboration of knowledge users, including patient engagement, to enhance its adoption and address real-world problems.
{"title":"Two decades of network meta-analysis: Roadmap to their applications and challenges","authors":"Areti Angeliki Veroniki, Ivan Florez, Brian Hutton, Sharon E. Straus, Andrea C. Tricco","doi":"10.1002/jrsm.1744","DOIUrl":"10.1002/jrsm.1744","url":null,"abstract":"<p>Recently, Ades and colleagues discussed the controversies and advancements in network meta-analysis (NMA) over the past two decades, discussing its reliability, assumptions, novel approaches, and provided some useful recommendations for the conduction of NMAs. The present discussion paper builds on the insights by Ades and colleagues, providing a roadmap for NMA applications, advancements in software and tools, and approaches designed to facilitate the assessment and interpretation of NMA findings. It also discusses the impact of NMA across disciplines, particularly for policymakers and guideline developers. Despite 20 years of NMA history, challenges remain in understanding and assessing assumptions, communicating and interpreting findings, and applying common approaches like network meta-regression and NMA involving non-randomized studies in readily available software. NMA has proven particularly valuable in clinical decision-making, which highlights the need for additional training and interdisciplinary collaboration of knowledge users, including patient engagement, to enhance its adoption and address real-world problems.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jrsm.1744","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141854315","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dan Jackson, Landan Zhang, Robert Hettle, Miranda Cooper
We respond to some of the methodological issues raised in a recent review of network meta-analysis (NMA). We also provide a health technology developer's perspective and consider the future application of NMA to health technology assessment.
{"title":"‘Twenty years of network meta-analysis: Continuing controversies and recent developments’: A health technology assessment perspective","authors":"Dan Jackson, Landan Zhang, Robert Hettle, Miranda Cooper","doi":"10.1002/jrsm.1740","DOIUrl":"10.1002/jrsm.1740","url":null,"abstract":"<p>We respond to some of the methodological issues raised in a recent review of network meta-analysis (NMA). We also provide a health technology developer's perspective and consider the future application of NMA to health technology assessment.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-07-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141791489","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. E. Ades, Nicky J. Welton, Sofia Dias, Deborah M. Caldwell, David M. Phillippo
We respond to discussant comments on our paper “Twenty years of network meta-analysis: Continuing controversies and recent developments” (https://doi.org/10.1002/jrsm.1700) and raise some additional points for consideration, including: the way in which methodological guidance is generated; integration of the estimand framework with evidence synthesis; and implications of the European Joint Clinical Assessment. We ask: what properties are required of population adjustment methods to enable transparent and consistent decision-making? We also ask why individual patient data is not routinely made available to re-imbursement authorities and clinical guideline developers.
{"title":"Response to discussant comments on “NMA, the first 20 years”","authors":"A. E. Ades, Nicky J. Welton, Sofia Dias, Deborah M. Caldwell, David M. Phillippo","doi":"10.1002/jrsm.1745","DOIUrl":"10.1002/jrsm.1745","url":null,"abstract":"<p>We respond to discussant comments on our paper “<i>Twenty years of network meta-analysis: Continuing controversies and recent developments</i>” (https://doi.org/10.1002/jrsm.1700) and raise some additional points for consideration, including: the way in which methodological guidance is generated; integration of the estimand framework with evidence synthesis; and implications of the European Joint Clinical Assessment. We ask: what properties are required of population adjustment methods to enable transparent and consistent decision-making? We also ask why individual patient data is not routinely made available to re-imbursement authorities and clinical guideline developers.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-07-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jrsm.1745","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141756063","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This discussion contribution provides some subjective early history of network meta-analysis and also proposes a new bipartite graph structure to better represent multi-arm trials.
本讨论提供了一些网络荟萃分析的早期主观历史,还提出了一种新的双方图结构,以更好地表示多臂试验。
{"title":"Network meta-analysis: Looping back","authors":"Thomas Lumley","doi":"10.1002/jrsm.1743","DOIUrl":"10.1002/jrsm.1743","url":null,"abstract":"<p>This discussion contribution provides some subjective early history of network meta-analysis and also proposes a new bipartite graph structure to better represent multi-arm trials.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jrsm.1743","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141756062","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Michal Shimonovich, Hilary Thomson, Anna Pearce, Srinivasa Vittal Katikireddi
BackgroundBradford Hill (BH) viewpoints are widely used to assess causality in systematic reviews, but their application has often lacked reproducibility. We describe an approach for assessing causality within systematic reviews (‘causal’ reviews), illustrating its application to the topic of income inequality and health. Our approach draws on principles of process tracing, a method used for case study research, to harness BH viewpoints to judge evidence for causal claims.MethodsIn process tracing, a hypothesis may be confirmed by observing highly unique evidence and disconfirmed by observing highly definitive evidence. We drew on these principles to consider the value of finding supportive or contradictory evidence for each BH viewpoint characterised by its uniqueness and definitiveness.ResultsIn our exemplar systematic review, we hypothesised that income inequality adversely affects self‐rated health and all‐cause mortality. BH viewpoints ‘analogy’ and ‘coherence’ were excluded from the causal assessment because of their low uniqueness and low definitiveness. The ‘experiment’ viewpoint was considered highly unique and highly definitive, and thus could be particularly valuable. We propose five steps for using BH viewpoints in a ‘causal’ review: (1) define the hypothesis; (2) characterise each viewpoint; (3) specify the evidence expected for each BH viewpoint for a true or untrue hypothesis; (4) gather evidence for each viewpoint (e.g., systematic review meta‐analyses, critical appraisal, background knowledge); (5) consider if each viewpoint was met (supportive evidence) or unmet (contradictory evidence).ConclusionsIncorporating process tracing has the potential to provide transparency and structure when using BH viewpoints in ‘causal’ reviews.
{"title":"Applying Bradford Hill to assessing causality in systematic reviews: A transparent approach using process tracing","authors":"Michal Shimonovich, Hilary Thomson, Anna Pearce, Srinivasa Vittal Katikireddi","doi":"10.1002/jrsm.1730","DOIUrl":"https://doi.org/10.1002/jrsm.1730","url":null,"abstract":"BackgroundBradford Hill (BH) viewpoints are widely used to assess causality in systematic reviews, but their application has often lacked reproducibility. We describe an approach for assessing causality within systematic reviews (‘causal’ reviews), illustrating its application to the topic of income inequality and health. Our approach draws on principles of process tracing, a method used for case study research, to harness BH viewpoints to judge evidence for causal claims.MethodsIn process tracing, a hypothesis may be confirmed by observing highly unique evidence and disconfirmed by observing highly definitive evidence. We drew on these principles to consider the value of finding supportive or contradictory evidence for each BH viewpoint characterised by its uniqueness and definitiveness.ResultsIn our exemplar systematic review, we hypothesised that income inequality adversely affects self‐rated health and all‐cause mortality. BH viewpoints ‘analogy’ and ‘coherence’ were excluded from the causal assessment because of their low uniqueness and low definitiveness. The ‘experiment’ viewpoint was considered highly unique and highly definitive, and thus could be particularly valuable. We propose five steps for using BH viewpoints in a ‘causal’ review: (1) define the hypothesis; (2) characterise each viewpoint; (3) specify the evidence expected for each BH viewpoint for a true or untrue hypothesis; (4) gather evidence for each viewpoint (e.g., systematic review meta‐analyses, critical appraisal, background knowledge); (5) consider if each viewpoint was met (supportive evidence) or unmet (contradictory evidence).ConclusionsIncorporating process tracing has the potential to provide transparency and structure when using BH viewpoints in ‘causal’ reviews.","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":9.8,"publicationDate":"2024-06-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141508855","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Amanda Konet, Ian Thomas, Gerald Gartlehner, Leila Kahwati, Rainer Hilscher, Shannon Kugley, Karen Crotty, Meera Viswanathan, Robert Chew
Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT-4) for extracting pre-specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT-4. GPT-4 required use of a third-party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT-4 with the plug-in was lower (68.8%); however, most of the errors were due to the plug-in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT-4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open-source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.
准确的数据提取是证据合成的关键组成部分,也是获得有效结果的关键。可公开获取的大型语言模型(LLM)的出现引起了人们对这些证据综合工具的兴趣,同时也为选择 LLM 带来了不确定性。我们比较了两种广泛使用的 LLM(Claude 2 和 GPT-4)在从先前完成的系统综述中收录的 10 篇已发表文章中提取预先指定的数据元素时的性能。我们使用提示和完整的研究 PDF 来比较 Claude 2 和 GPT-4 浏览器版本的输出结果。GPT-4 需要使用第三方插件来上传和解析 PDF。Claude 2 的准确率很高(96.3%)。使用插件的 GPT-4 的准确率较低(68.8%);不过,大部分错误是由插件造成的。两种 LLM 都能正确识别源 PDF 中缺少预先指定的数据元素,并为文章中未明确报告的数据元素生成正确的信息。二次分析表明,当提供 PDF 中的选定文本时,Claude 2 和 GPT-4 分别准确提取了 98.7% 和 100% 的数据元素。局限性包括:使用的研究 PDF 范围较窄;仅使用 Claude 2 完成了提示开发;我们无法保证开源文章未被用于训练 LLM。这项研究凸显了 LLM 在数据提取方面的革命性潜力,但同时也强调了精确 PDF 解析的重要性。目前,人类研究人员仍然有必要对 LLM 提取进行验证。
{"title":"Performance of two large language models for data extraction in evidence synthesis","authors":"Amanda Konet, Ian Thomas, Gerald Gartlehner, Leila Kahwati, Rainer Hilscher, Shannon Kugley, Karen Crotty, Meera Viswanathan, Robert Chew","doi":"10.1002/jrsm.1732","DOIUrl":"10.1002/jrsm.1732","url":null,"abstract":"<p>Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT-4) for extracting pre-specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT-4. GPT-4 required use of a third-party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT-4 with the plug-in was lower (68.8%); however, most of the errors were due to the plug-in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT-4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open-source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-06-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141417088","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Citation indices providing information on backward citation (BWC) and forward citation (FWC) links are essential for literature discovery, bibliographic analysis, and knowledge synthesis, especially when language barriers impede document identification. However, the suitability of citation indices varies. While some have been analyzed, the majority, whether new or established, lack comprehensive evaluation. Therefore, this study evaluates the citation coverage of the citation indices of 59 databases, encompassing the widely used Google Scholar, Scopus, and Web of Science alongside many others never previously analyzed, such as the emerging Lens, Scite, Dimensions, and OpenAlex or the subject-specific PubMed and JSTOR. Through a comprehensive analysis using 259 journal articles from across disciplines, this research aims to guide scholars in selecting indices with broader document coverage and more accurate and comprehensive backward and forward citation links. Key findings highlight Google Scholar, ResearchGate, Semantic Scholar, and Lens as leading options for FWC searching, with Lens providing superior download capabilities. For BWC searching, the Web of Science Core Collection can be recommended over Scopus for accuracy. BWC information from publisher databases such as IEEE Xplore or ScienceDirect was generally found to be the most accurate, yet only available for a limited number of articles. The findings will help scholars conducting systematic reviews, meta-analyses, and bibliometric analyses to select the most suitable databases for citation searching.
提供后向引文(BWC)和前向引文(FWC)链接信息的引文索引对于文献发现、书目分析和知识合成至关重要,尤其是在语言障碍阻碍文献识别的情况下。然而,引文索引的适用性各不相同。虽然对一些索引进行了分析,但大多数索引,无论是新的还是已建立的,都缺乏全面的评估。因此,本研究评估了 59 个数据库的引文索引的引文覆盖范围,其中包括广泛使用的 Google Scholar、Scopus 和 Web of Science,以及许多以前从未分析过的其他数据库,如新兴的 Lens、Scite、Dimensions 和 OpenAlex 或特定主题的 PubMed 和 JSTOR。本研究通过对 259 篇跨学科期刊论文进行全面分析,旨在指导学者选择文献覆盖面更广、前后引文链接更准确、更全面的索引。主要研究结果表明,Google Scholar、ResearchGate、Semantic Scholar 和 Lens 是 FWC 搜索的主要选择,其中 Lens 的下载功能更胜一筹。在 BWC 搜索方面,推荐使用 Web of Science 核心合集,其准确性优于 Scopus。一般认为,IEEE Xplore 或 ScienceDirect 等出版商数据库中的 BWC 信息最为准确,但只能提供有限数量的文章。这些发现将有助于进行系统综述、荟萃分析和文献计量学分析的学者选择最合适的数据库进行引文检索。
{"title":"Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices","authors":"Michael Gusenbauer","doi":"10.1002/jrsm.1729","DOIUrl":"10.1002/jrsm.1729","url":null,"abstract":"<p>Citation indices providing information on backward citation (BWC) and forward citation (FWC) links are essential for literature discovery, bibliographic analysis, and knowledge synthesis, especially when language barriers impede document identification. However, the suitability of citation indices varies. While some have been analyzed, the majority, whether new or established, lack comprehensive evaluation. Therefore, this study evaluates the citation coverage of the citation indices of 59 databases, encompassing the widely used Google Scholar, Scopus, and Web of Science alongside many others never previously analyzed, such as the emerging Lens, Scite, Dimensions, and OpenAlex or the subject-specific PubMed and JSTOR. Through a comprehensive analysis using 259 journal articles from across disciplines, this research aims to guide scholars in selecting indices with broader document coverage and more accurate and comprehensive backward and forward citation links. Key findings highlight Google Scholar, ResearchGate, Semantic Scholar, and Lens as leading options for FWC searching, with Lens providing superior download capabilities. For BWC searching, the Web of Science Core Collection can be recommended over Scopus for accuracy. BWC information from publisher databases such as IEEE Xplore or ScienceDirect was generally found to be the most accurate, yet only available for a limited number of articles. The findings will help scholars conducting systematic reviews, meta-analyses, and bibliometric analyses to select the most suitable databases for citation searching.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jrsm.1729","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141320051","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}