Zalaya Simmons, Beti Evans, Tamsyn Harris, Harry Woolnough, Lauren Dunn, Jonathon Fuller, Kerry Cella, Daphne Duval
Introduction
Data extraction is a critical but resource-intensive step of the evidence review process. Whilst there is evidence that artificial intelligence (AI) and large language models (LLMs) can improve the efficiency of data extraction from randomized controlled trials, their potential for other study designs is unclear. In this context, this study aimed to evaluate the performance of a bespoke LLM model pipeline (Retrieval-Augmented Generation pipeline utilizing LLaMa 3-70B) to automate data extraction from a range of study designs by assessing the accuracy and reliability of the extractions measured as error types and acceptability.
Methods
Accuracy was assessed by retrospectively comparing the LLM extractions against human extractions from a review previously conducted by the authors. A total of 173 data fields from 24 articles (including experimental, observational, qualitative, and modeling studies) were assessed, of which three were used for prompt engineering. Reliability was assessed by calculating the mean maximum agreement rate (the highest proportion of identical returns from 10 consecutive extractions) for 116 data fields from 16 of the 24 studies. An evaluation framework was developed to assess the accuracy and reliability of LLM outputs measured as error types and acceptability (acceptability was assessed on whether it would be usable in real-world settings if the model acted as one reviewer and a human as a second reviewer).
Results
Of the 173 data fields evaluated for accuracy, 68% were rated by human reviewers as acceptable (consistent with what is deemed to be acceptable data extraction from a human reviewer). However, acceptability ratings varied depending on the data field extracted (33% to 100%), with at least 90% acceptability for “objective,” “setting,” and “study design,” but 54% or less for data fields such as “outcome” and “time period.” For reliability, the mean maximum agreement rate was 0.71 (SD: 0.28), with variation across different data fields.
Conclusion
This evaluation demonstrates the potential for LLMs, when paired with human quality assurance, to support data extraction in evidence reviews that include a range of study designs. However, further improvements in performance and validation are required before the model can be introduced into review workflows.
{"title":"Assessing the Feasibility and Acceptability of a Bespoke Large Language Model Pipeline to Extract Data From Different Study Designs for Public Health Evidence Reviews","authors":"Zalaya Simmons, Beti Evans, Tamsyn Harris, Harry Woolnough, Lauren Dunn, Jonathon Fuller, Kerry Cella, Daphne Duval","doi":"10.1002/cesm.70061","DOIUrl":"10.1002/cesm.70061","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>Data extraction is a critical but resource-intensive step of the evidence review process. Whilst there is evidence that artificial intelligence (AI) and large language models (LLMs) can improve the efficiency of data extraction from randomized controlled trials, their potential for other study designs is unclear. In this context, this study aimed to evaluate the performance of a bespoke LLM model pipeline (Retrieval-Augmented Generation pipeline utilizing LLaMa 3-70B) to automate data extraction from a range of study designs by assessing the accuracy and reliability of the extractions measured as error types and acceptability.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>Accuracy was assessed by retrospectively comparing the LLM extractions against human extractions from a review previously conducted by the authors. A total of 173 data fields from 24 articles (including experimental, observational, qualitative, and modeling studies) were assessed, of which three were used for prompt engineering. Reliability was assessed by calculating the mean maximum agreement rate (the highest proportion of identical returns from 10 consecutive extractions) for 116 data fields from 16 of the 24 studies. An evaluation framework was developed to assess the accuracy and reliability of LLM outputs measured as error types and acceptability (acceptability was assessed on whether it would be usable in real-world settings if the model acted as one reviewer and a human as a second reviewer).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Of the 173 data fields evaluated for accuracy, 68% were rated by human reviewers as acceptable (consistent with what is deemed to be acceptable data extraction from a human reviewer). However, acceptability ratings varied depending on the data field extracted (33% to 100%), with at least 90% acceptability for “objective,” “setting,” and “study design,” but 54% or less for data fields such as “outcome” and “time period.” For reliability, the mean maximum agreement rate was 0.71 (SD: 0.28), with variation across different data fields.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>This evaluation demonstrates the potential for LLMs, when paired with human quality assurance, to support data extraction in evidence reviews that include a range of study designs. However, further improvements in performance and validation are required before the model can be introduced into review workflows.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12584109/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145454519","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jennifer Petkovic, Jordi Pardo Pardo, Vivian Welch, Omar Dewidar, Lara J. Maxwell, Andrea Darzi, Tamara Lotfi, Lawrence Mbuagbaw, Kevin Pottie, Peter Tugwell
This tutorial focuses on how to get started with considering health equity in systematic reviews of interventions. We will explain why health equity should be considered, how to frame your question, and which interest-holders to engage. This is the first tutorial in a series on health equity. The second tutorial focuses on implementing health equity methods in your review.
{"title":"Health Equity in Systematic Reviews: A Tutorial—Part 1 Getting Started With Health Equity in Your Review","authors":"Jennifer Petkovic, Jordi Pardo Pardo, Vivian Welch, Omar Dewidar, Lara J. Maxwell, Andrea Darzi, Tamara Lotfi, Lawrence Mbuagbaw, Kevin Pottie, Peter Tugwell","doi":"10.1002/cesm.70055","DOIUrl":"https://doi.org/10.1002/cesm.70055","url":null,"abstract":"<p>This tutorial focuses on how to get started with considering health equity in systematic reviews of interventions. We will explain why health equity should be considered, how to frame your question, and which interest-holders to engage. This is the first tutorial in a series on health equity. The second tutorial focuses on implementing health equity methods in your review.</p>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70055","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145406969","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Molly Murton, Ellie Boulton, Shona Cross, Ambar Khan, Swati Kumar, Giuseppina Magri, Charlotte Marris, David Slater, Emma Worthington, Elizabeth Lunn
Introduction
Systematic literature reviews (SLRs) of randomized clinical trials (RCTs) underpin evidence-based medicine but can be limited by the intensive resource demands of data extraction. Recent advances in accessible large-language models (LLMs) hold promise for automating this step, however testing is limited across different outcomes and disease areas.
Methods
This study developed prompt engineering strategies for GPT-4o to extract data from RCTs across three disease areas: non-small cell lung cancer, endometrial cancer and hypertrophic cardiomyopathy. Prompts were iteratively refined during the development phase, then tested on unseen data. Performance was evaluated via comparison to human extraction of the same data, using F1 scores, precision, recall and percentage accuracy.
Results
The LLM was highly effective for extracting study and baseline characteristics, often equaling human performance, with test F1 scores exceeding 0.85. Complex efficacy and adverse event data proved more challenging, with test F1 scores ranging from 0.22 to 0.50. Transferability of prompts across disease areas was promising but varied, highlighting the need for disease-specific refinement.
Conclusion
Our findings demonstrate the potential of LLMs, guided by rigorous prompt engineering, to augment the SLR process. However, human oversight remains essential, particularly for complex and nuanced data. As these technologies evolve, continued validation of AI tools will be necessary to ensure accuracy and reliability, and safeguarding of the quality of evidence synthesis.
{"title":"Harnessing Large-Language Models for Efficient Data Extraction in Systematic Reviews: The Role of Prompt Engineering","authors":"Molly Murton, Ellie Boulton, Shona Cross, Ambar Khan, Swati Kumar, Giuseppina Magri, Charlotte Marris, David Slater, Emma Worthington, Elizabeth Lunn","doi":"10.1002/cesm.70058","DOIUrl":"10.1002/cesm.70058","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>Systematic literature reviews (SLRs) of randomized clinical trials (RCTs) underpin evidence-based medicine but can be limited by the intensive resource demands of data extraction. Recent advances in accessible large-language models (LLMs) hold promise for automating this step, however testing is limited across different outcomes and disease areas.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>This study developed prompt engineering strategies for GPT-4o to extract data from RCTs across three disease areas: non-small cell lung cancer, endometrial cancer and hypertrophic cardiomyopathy. Prompts were iteratively refined during the development phase, then tested on unseen data. Performance was evaluated via comparison to human extraction of the same data, using F1 scores, precision, recall and percentage accuracy.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The LLM was highly effective for extracting study and baseline characteristics, often equaling human performance, with test F1 scores exceeding 0.85. Complex efficacy and adverse event data proved more challenging, with test F1 scores ranging from 0.22 to 0.50. Transferability of prompts across disease areas was promising but varied, highlighting the need for disease-specific refinement.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>Our findings demonstrate the potential of LLMs, guided by rigorous prompt engineering, to augment the SLR process. However, human oversight remains essential, particularly for complex and nuanced data. As these technologies evolve, continued validation of AI tools will be necessary to ensure accuracy and reliability, and safeguarding of the quality of evidence synthesis.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12559671/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145403637","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kevin M. Kallmes, Jade Thurnham, Marius Sauca, Ranita Tarchand, Keith R. Kallmes, Karl J. Holub
Introduction
While artificial intelligence (AI) tools have been utilized for individual stages within the systematic literature review (SLR) process, no tool has previously been shown to support each critical SLR step. In addition, the need for expert oversight has been recognized to ensure the quality of SLR findings. Here, we describe a complete methodology for utilizing our AI SLR tool with human-in-the-loop curation workflows, as well as AI validations, time savings, and approaches to ensure compliance with best review practices.
Methods
SLRs require completing Search, Screening, and Extraction from relevant studies, with meta-analysis and critical appraisal as relevant. We present a full methodological framework for completing SLRs utilizing our AutoLit software (Nested Knowledge). This system integrates AI models into the central steps in SLR: Search strategy generation, Dual Screening of Titles/Abstracts and Full Texts, and Extraction of qualitative and quantitative evidence. The system also offers manual Critical Appraisal and Insight drafting and fully-automated Network Meta-analysis. Validations comparing AI performance to experts are reported, and where relevant, time savings and ‘rapid review’ alternatives to the SLR workflow.
Results
Search strategy generation with the Smart Search AI can turn a Research Question into full Boolean strings with 76.8% and 79.6% Recall in two validation sets. Supervised machine learning tools can achieve 82–97% Recall in reviewer-level Screening. Population, Interventions/Comparators, and Outcomes (PICOs) extraction achieved F1 of 0.74; accuracy for study type, location, and size were 74%, 78%, and 91%, respectively. Time savings of 50% in Abstract Screening and 70–80% in qualitative extraction were reported. Extraction of user-specified qualitative and quantitative tags and data elements remains exploratory and requires human curation for SLRs.
Conclusion
AI systems can support high-quality, human-in-the-loop execution of key SLR stages. Transparency, replicability, and expert oversight are central to the use of AI SLR tools.
{"title":"Human-in-the-Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software","authors":"Kevin M. Kallmes, Jade Thurnham, Marius Sauca, Ranita Tarchand, Keith R. Kallmes, Karl J. Holub","doi":"10.1002/cesm.70059","DOIUrl":"https://doi.org/10.1002/cesm.70059","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>While artificial intelligence (AI) tools have been utilized for individual stages within the systematic literature review (SLR) process, no tool has previously been shown to support each critical SLR step. In addition, the need for expert oversight has been recognized to ensure the quality of SLR findings. Here, we describe a complete methodology for utilizing our AI SLR tool with human-in-the-loop curation workflows, as well as AI validations, time savings, and approaches to ensure compliance with best review practices.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>SLRs require completing Search, Screening, and Extraction from relevant studies, with meta-analysis and critical appraisal as relevant. We present a full methodological framework for completing SLRs utilizing our AutoLit software (Nested Knowledge). This system integrates AI models into the central steps in SLR: Search strategy generation, Dual Screening of Titles/Abstracts and Full Texts, and Extraction of qualitative and quantitative evidence. The system also offers manual Critical Appraisal and Insight drafting and fully-automated Network Meta-analysis. Validations comparing AI performance to experts are reported, and where relevant, time savings and ‘rapid review’ alternatives to the SLR workflow.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Search strategy generation with the Smart Search AI can turn a Research Question into full Boolean strings with 76.8% and 79.6% Recall in two validation sets. Supervised machine learning tools can achieve 82–97% Recall in reviewer-level Screening. Population, Interventions/Comparators, and Outcomes (PICOs) extraction achieved F1 of 0.74; accuracy for study type, location, and size were 74%, 78%, and 91%, respectively. Time savings of 50% in Abstract Screening and 70–80% in qualitative extraction were reported. Extraction of user-specified qualitative and quantitative tags and data elements remains exploratory and requires human curation for SLRs.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>AI systems can support high-quality, human-in-the-loop execution of key SLR stages. Transparency, replicability, and expert oversight are central to the use of AI SLR tools.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70059","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145367149","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Bing Hu, Emmalie Tomini, Tricia Corrin, Kusala Pussegoda, Elias Sandner, Andre Henriques, Alice Simniceanu, Luca Fontana, Andreas Wagner, Stephanie Brazeau, Lisa Waddell
Background
Public health events of international concern highlight the need for up-to-date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic-AI assisted pipeline (GREP-Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs).
Methods
In this study, the performance of the GREP-Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o-mini, Llama3.1, and Phi4). The GREP-Agent framework integrates multiple LLMs and human feedback to fine-tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic-AI system is presented and measured by accuracy, precision, recall, and F1-score metrics.
Results
The results show how each phase of the GREP-Agent system incrementally improves accuracy regardless of the LLM. We found that GREP-Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%–88.9% after fine-tuning and to 86.4%–95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies.
Conclusions
The GREP-Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human-in-the-loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human-centric approach.
{"title":"Enhancing Evidence Synthesis Efficiency: Leveraging Large Language Models and Agentic Workflows for Optimized Literature Screening","authors":"Bing Hu, Emmalie Tomini, Tricia Corrin, Kusala Pussegoda, Elias Sandner, Andre Henriques, Alice Simniceanu, Luca Fontana, Andreas Wagner, Stephanie Brazeau, Lisa Waddell","doi":"10.1002/cesm.70042","DOIUrl":"10.1002/cesm.70042","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Public health events of international concern highlight the need for up-to-date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic-AI assisted pipeline (GREP-Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>In this study, the performance of the GREP-Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o-mini, Llama3.1, and Phi4). The GREP-Agent framework integrates multiple LLMs and human feedback to fine-tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic-AI system is presented and measured by accuracy, precision, recall, and F1-score metrics.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The results show how each phase of the GREP-Agent system incrementally improves accuracy regardless of the LLM. We found that GREP-Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%–88.9% after fine-tuning and to 86.4%–95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>The GREP-Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human-in-the-loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human-centric approach.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12538819/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145351061","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jennifer Petkovic, Jordi Pardo Pardo, Vivian Welch, Omar Dewidar, Lara J. Maxwell, Andrea Darzi, Tamara Lotfi, Lawrence Mbuagbaw, Kevin Pottie, Peter Tugwell
This is the second and final tutorial in a series on health equity. It provides detailed guidance for considering health equity in systematic reviews of interventions. We will explain how to include and report health equity in all remaining sections of the review.
{"title":"Health Equity in Systematic Reviews: A Tutorial—Part 2 Implementing Health Equity Throughout Your Methods","authors":"Jennifer Petkovic, Jordi Pardo Pardo, Vivian Welch, Omar Dewidar, Lara J. Maxwell, Andrea Darzi, Tamara Lotfi, Lawrence Mbuagbaw, Kevin Pottie, Peter Tugwell","doi":"10.1002/cesm.70054","DOIUrl":"10.1002/cesm.70054","url":null,"abstract":"<p>This is the second and final tutorial in a series on health equity. It provides detailed guidance for considering health equity in systematic reviews of interventions. We will explain how to include and report health equity in all remaining sections of the review.</p>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12522172/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145310506","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Anas Elmahi, Nathalie Doolan, Mohiedin Hezima, Anwar Gowey, Daragh Moneley, Seamus McHugh, Sayed Aly, Peter Naughton, Elrasheid A. H. Kheirelseid
<div> <section> <h3> Background</h3> <p>Intermittent claudication (IC) is a hallmark symptom of peripheral arterial disease (PAD), causing pain and discomfort during physical activity caused by reduced blood flow to the lower extremities. The condition significantly impairs mobility and quality of life (QoL) in affected individuals. Treatment options for IC range from conservative approaches, including best medical therapy (BMT) and supervised exercise therapy (SET), to invasive interventions like angioplasty and open re-vascularization.</p> </section> <section> <h3> Aim</h3> <p>This meta-analysis and systematic review seek to assess the long-term results of invasive procedures concerning Noninvasive treatments for the management of patients with IC.</p> </section> <section> <h3> Methods</h3> <p>A comprehensive search was conducted in October 2024 across databases containing PubMed, MEDLINE, Cochrane Library, Embase, and Scopus. Randomized controlled trials (RCTs) comparing invasive interventions to Noninvasive treatments were included. Primary outcomes were quality of life (QoL), ankle-brachial pressure index (ABPI), and maximum walking distance (MWD). Secondary outcomes were major adverse cardiovascular events (MACE), mortality, complications, and re-intervention rates. Data analysis was conducted using the Cochrane Review Manager 5. Follow-up duration was between 2 and 7 years, longest available between 2 and 7 years; prioritized 2 years when present.</p> </section> <section> <h3> Results</h3> <p>A total of 11 RCTs with 1379 patients were included in the analysis. Invasive treatments demonstrated a significant improvement in MWD and ABPI compared to Noninvasive treatments (MWD pooled Mean Difference (MD) = 64.94 [10.77, 115.12] 95% CI, <i>p</i> = .02, 5 studies, and ABPI pooled MD = 0.15 [0.04, 0.26] 95% CI, <i>p</i> = .006, 5 studies). However, invasive interventions were associated with a higher rate of complications, including increased amputation risk (Pooled odds ratio (OR) = 2.46 [0.44, 13.94] 95% CI, <i>p</i> = .31, 3 studies), though this was not statistically significant. Long-term rates were higher in the Noninvasive treatment group (Pooled OR: 0.56 [0.33, 0.97] 95% CI, <i>p</i> = .04).</p> </section> <section> <h3> Conclusions</h3> <p>Both invasive and Noninvasive treatments are effective in managing IC. Invasive treatments provide greater improvement in blood flow and walking distance, but the risk of complications and re-interventio
{"title":"Long-Term Outcomes of Invasive vs Noninvasive Treatment for Intermittent Claudication: A Systematic Review and Meta-Analysis","authors":"Anas Elmahi, Nathalie Doolan, Mohiedin Hezima, Anwar Gowey, Daragh Moneley, Seamus McHugh, Sayed Aly, Peter Naughton, Elrasheid A. H. Kheirelseid","doi":"10.1002/cesm.70053","DOIUrl":"https://doi.org/10.1002/cesm.70053","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Intermittent claudication (IC) is a hallmark symptom of peripheral arterial disease (PAD), causing pain and discomfort during physical activity caused by reduced blood flow to the lower extremities. The condition significantly impairs mobility and quality of life (QoL) in affected individuals. Treatment options for IC range from conservative approaches, including best medical therapy (BMT) and supervised exercise therapy (SET), to invasive interventions like angioplasty and open re-vascularization.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Aim</h3>\u0000 \u0000 <p>This meta-analysis and systematic review seek to assess the long-term results of invasive procedures concerning Noninvasive treatments for the management of patients with IC.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>A comprehensive search was conducted in October 2024 across databases containing PubMed, MEDLINE, Cochrane Library, Embase, and Scopus. Randomized controlled trials (RCTs) comparing invasive interventions to Noninvasive treatments were included. Primary outcomes were quality of life (QoL), ankle-brachial pressure index (ABPI), and maximum walking distance (MWD). Secondary outcomes were major adverse cardiovascular events (MACE), mortality, complications, and re-intervention rates. Data analysis was conducted using the Cochrane Review Manager 5. Follow-up duration was between 2 and 7 years, longest available between 2 and 7 years; prioritized 2 years when present.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>A total of 11 RCTs with 1379 patients were included in the analysis. Invasive treatments demonstrated a significant improvement in MWD and ABPI compared to Noninvasive treatments (MWD pooled Mean Difference (MD) = 64.94 [10.77, 115.12] 95% CI, <i>p</i> = .02, 5 studies, and ABPI pooled MD = 0.15 [0.04, 0.26] 95% CI, <i>p</i> = .006, 5 studies). However, invasive interventions were associated with a higher rate of complications, including increased amputation risk (Pooled odds ratio (OR) = 2.46 [0.44, 13.94] 95% CI, <i>p</i> = .31, 3 studies), though this was not statistically significant. Long-term rates were higher in the Noninvasive treatment group (Pooled OR: 0.56 [0.33, 0.97] 95% CI, <i>p</i> = .04).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>Both invasive and Noninvasive treatments are effective in managing IC. Invasive treatments provide greater improvement in blood flow and walking distance, but the risk of complications and re-interventio","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70053","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145224118","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Elicit AI aims to simplify and accelerate the systematic review process without compromising accuracy. However, research on Elicit's performance is limited.
Objectives
To determine whether Elicit AI is a viable tool for systematic literature searches and title/abstract screening stages.
Methods
We compared the included studies in four evidence syntheses to those identified using the subscription-based version of Elicit Pro in Review mode. We calculated sensitivity, precision and observed patterns in the performance of Elicit.
Results
The sensitivity of Elicit was poor, averaging 39.5% (25.5–69.2%) compared to 94.5% (91.1–98.0%) in the original reviews. However, Elicit identified some included studies not identified by the original searches and had an average of 41.8% precision (35.6–46.2%) which was higher than the 7.55% average of the original reviews (0.65–14.7%).
Discussion
At the time of this evaluation, Elicit did not search with high enough sensitivity to replace traditional literature searching. However, the high precision of searching in Elicit could prove useful for preliminary searches, and the unique studies identified mean that Elicit can be used by researchers as a useful adjunct.
Conclusion
Whilst Elicit searches are currently not sensitive enough to replace traditional searching, Elicit is continually improving, and further evaluations should be undertaken as new developments take place.
{"title":"Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies","authors":"Oscar Lau, Su Golder","doi":"10.1002/cesm.70050","DOIUrl":"https://doi.org/10.1002/cesm.70050","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Elicit AI aims to simplify and accelerate the systematic review process without compromising accuracy. However, research on Elicit's performance is limited.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Objectives</h3>\u0000 \u0000 <p>To determine whether Elicit AI is a viable tool for systematic literature searches and title/abstract screening stages.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>We compared the included studies in four evidence syntheses to those identified using the subscription-based version of Elicit Pro in Review mode. We calculated sensitivity, precision and observed patterns in the performance of Elicit.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The sensitivity of Elicit was poor, averaging 39.5% (25.5–69.2%) compared to 94.5% (91.1–98.0%) in the original reviews. However, Elicit identified some included studies not identified by the original searches and had an average of 41.8% precision (35.6–46.2%) which was higher than the 7.55% average of the original reviews (0.65–14.7%).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Discussion</h3>\u0000 \u0000 <p>At the time of this evaluation, Elicit did not search with high enough sensitivity to replace traditional literature searching. However, the high precision of searching in Elicit could prove useful for preliminary searches, and the unique studies identified mean that Elicit can be used by researchers as a useful adjunct.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>Whilst Elicit searches are currently not sensitive enough to replace traditional searching, Elicit is continually improving, and further evaluations should be undertaken as new developments take place.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70050","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145172063","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This tutorial focuses on split body trials in the context of a systematic review and meta-analysis. We will explain what split body trials are, the potential unit of analysis issues they can cause, and how to include data from split body trials in a systematic review.
{"title":"Split Body Trials in Systematic Reviews and Meta-Analyses: A Tutorial","authors":"Nuala Livingstone, Kerry Dwan, Marty Chaplin","doi":"10.1002/cesm.70052","DOIUrl":"https://doi.org/10.1002/cesm.70052","url":null,"abstract":"<p>This tutorial focuses on split body trials in the context of a systematic review and meta-analysis. We will explain what split body trials are, the potential unit of analysis issues they can cause, and how to include data from split body trials in a systematic review.</p><p>Split body trials micro learning module\u0000 \u0000 <figure>\u0000 <div><picture>\u0000 <source></source></picture><p></p>\u0000 </div>\u0000 </figure></p>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 5","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70052","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145146377","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xing Xing, Lifeng Lin, Mohammad Hassan Murad, Jiayi Tong
Introduction
Publication bias (PB) threatens the validity of meta-analyses by distorting effect size estimates, potentially leading to misleading conclusions. With advanced pattern recognition and multimodal capabilities, large language models (LLMs) may be able to evaluate PB and make the systematic review process more efficient.
Methods
We evaluated the ability of two state-of-the-art multimodal LLMs, GPT-4o and Llama 3.2 Vision, to detect PB using funnel plots alone and in combination with quantitative inputs. We simulated meta-analyses under varying conditions, including the absence of PB, different levels of presence of PB, varying total number of studies within a meta-analysis, and differing degrees of between-study heterogeneity.
Results
Neither GPT-4o nor Llama 3.2 Vision consistently detected the presence of PB across various settings. Under no-publication-bias conditions, GPT-4o achieved a higher specificity outperforming Llama 3.2 Vision, with the difference most shown in the meta-analyses with 20 or more studies. The inclusion of quantitative inputs alongside funnel plots did not significantly improve performance. Additionally, between-study heterogeneity and patterns of non-reported studies had minimal impact on the models’ assessments.
Conclusions
The ability of LLMs to detect PB without fine-tuning is limited at the present time. This study highlights the need for specialized model adaptation before LLMs can be effectively integrated into meta-analysis workflows. Future research can focus on targeted refinements to enhance LLM performance and utility in evidence synthesis.
{"title":"Leveraging AI for Meta-Analysis: Evaluating LLMs in Detecting Publication Bias for Next-Generation Evidence Synthesis","authors":"Xing Xing, Lifeng Lin, Mohammad Hassan Murad, Jiayi Tong","doi":"10.1002/cesm.70047","DOIUrl":"https://doi.org/10.1002/cesm.70047","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>Publication bias (PB) threatens the validity of meta-analyses by distorting effect size estimates, potentially leading to misleading conclusions. With advanced pattern recognition and multimodal capabilities, large language models (LLMs) may be able to evaluate PB and make the systematic review process more efficient.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>We evaluated the ability of two state-of-the-art multimodal LLMs, GPT-4o and Llama 3.2 Vision, to detect PB using funnel plots alone and in combination with quantitative inputs. We simulated meta-analyses under varying conditions, including the absence of PB, different levels of presence of PB, varying total number of studies within a meta-analysis, and differing degrees of between-study heterogeneity.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Neither GPT-4o nor Llama 3.2 Vision consistently detected the presence of PB across various settings. Under no-publication-bias conditions, GPT-4o achieved a higher specificity outperforming Llama 3.2 Vision, with the difference most shown in the meta-analyses with 20 or more studies. The inclusion of quantitative inputs alongside funnel plots did not significantly improve performance. Additionally, between-study heterogeneity and patterns of non-reported studies had minimal impact on the models’ assessments.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>The ability of LLMs to detect PB without fine-tuning is limited at the present time. This study highlights the need for specialized model adaptation before LLMs can be effectively integrated into meta-analysis workflows. Future research can focus on targeted refinements to enhance LLM performance and utility in evidence synthesis.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 5","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70047","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145101484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号