Cochrane Evidence Synthesis and Methods最新文献

{"title":"Systematic Reviews as Part of Doctoral Theses and for the Promotion to Associate Professor: A Descriptive Study of University Policies in Sweden.","authors":"Martin Ringsten, Lea Styrmisdottir, Matilda Naesström, Minna Johansson, Matteo Bruschettini, Susanna M Wallerstedt","doi":"10.1002/cesm.70069","DOIUrl":"https://doi.org/10.1002/cesm.70069","url":null,"abstract":"<p><strong>Background: </strong>Almost a decade ago, about half of biomedical PhD programs across Europe specifically stated that systematic reviews could not be accepted as part of a doctoral thesis, illustrating limited merit value at that time. The aim of this study was to explore current Swedish university policies on this research design.</p><p><strong>Methods: </strong>Policy documents for PhD theses and applications to associate professor positions were obtained from all medical faculties at universities in Sweden. Instructions regarding systematic reviews, with focus on their merit value and related aspects, were independently extracted and categorized by two authors, with discrepancies resolved in consensus discussions.</p><p><strong>Results: </strong>All seven medical faculties accepted at least one systematic review within a PhD thesis, five restricted the number of such studies accepted, and five provided instructions regarding this study design. Regarding policies for promotion to associate professor, six medical faculties accepted at least one published systematic review to merit recognition-the remaining one required meta-analyses for acceptance-and three explicitly restricted the number of systematic reviews. No restrictions or guidance were provided for other designs intended to answer specific research questions.</p><p><strong>Conclusion: </strong>As of 2025, systematic reviews appear to be generally recognized as contributing to authors' academic merit. For this research design exclusively, some universities impose restrictions that may limit their recognition, and some provide guidance which may help ensure quality in reporting. These findings may encourage research to evaluate the merit value of systematic reviews in other settings, and to examine potential implications of restrictions and guidance in policy documents.</p>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"4 1","pages":"e70069"},"PeriodicalIF":0.0,"publicationDate":"2026-01-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12806540/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146000321","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Co-production Evaluation Tool Informed by Co-production Workshops for Use in Evidence Synthesis Contexts","authors":"Meena Khatwa,&nbsp;Vanessa Bennett,&nbsp;Rachael C. Edwards,&nbsp;Lisa Richardson,&nbsp;Phuong Tu Nguyen,&nbsp;Sajid Saleem,&nbsp;Sylvia Chaires,&nbsp;Alison O'Mara-Eves,&nbsp;Dylan Kneale","doi":"10.1002/cesm.70065","DOIUrl":"10.1002/cesm.70065","url":null,"abstract":"&lt;div&gt;\u0000 \u0000 \u0000 &lt;section&gt;\u0000 \u0000 &lt;h3&gt; Aim&lt;/h3&gt;\u0000 \u0000 &lt;p&gt;We aimed to co-produce a tool for evaluating co-production within evidence syntheses.&lt;/p&gt;\u0000 &lt;/section&gt;\u0000 \u0000 &lt;section&gt;\u0000 \u0000 &lt;h3&gt; Background&lt;/h3&gt;\u0000 \u0000 &lt;p&gt;Participatory approaches are recommended to enhance the salience and quality of evidence syntheses, and there is an increasing onus on co-producing evidence synthesis. Co-production is a way of working where research generators, beneficiaries and other interest holders work in equal partnership and for mutual benefit.&lt;/p&gt;\u0000 &lt;/section&gt;\u0000 \u0000 &lt;section&gt;\u0000 \u0000 &lt;h3&gt; Methods&lt;/h3&gt;\u0000 \u0000 &lt;div&gt;To develop our approach, we:\u0000\u0000 &lt;ul&gt;\u0000 \u0000 &lt;li&gt;\u0000 &lt;p&gt;Examined selected existing tools and frameworks that could be useful in evaluating co-production&lt;/p&gt;\u0000 &lt;/li&gt;\u0000 \u0000 &lt;li&gt;\u0000 &lt;p&gt;Developed an initial tool that was then modified through input from co-production workshops&lt;/p&gt;\u0000 &lt;/li&gt;\u0000 \u0000 &lt;li&gt;\u0000 &lt;p&gt;Piloted the tool and evaluation approach in a project as part of research involving co-producing a logic model to support evidence syntheses.&lt;/p&gt;\u0000 &lt;/li&gt;\u0000 &lt;/ul&gt;\u0000 &lt;/div&gt;\u0000 &lt;/section&gt;\u0000 \u0000 &lt;section&gt;\u0000 \u0000 &lt;h3&gt; Results&lt;/h3&gt;\u0000 \u0000 &lt;p&gt;The existing tools guidance and resources we examined were deemed to be oriented towards supporting the conduct and reporting of co-production, rather than evaluating what happens and how. This provided a basis for co-producing a new tool. A new tool was developed that captures our perspectives on: positionality and expertise; motivations and expected benefits; clarity of role and expectations; project involvement and contributions; value and recognition; skills, knowledge, and personal growth; relationships and networking; comfort, support, and accessibility; and decision-making and power sharing. We reflected that the tool and process for administering the tool worked well, and we liked the process of collective sensemaking.&lt;/p&gt;\u0000 &lt;/section&gt;\u0000 \u0000 &lt;section&gt;\u0000 \u0000 &lt;h3&gt; Conclusions&lt;/h3&gt;\u0000 \u0000 &lt;p&gt;We believe that the tool (which we refer to as the STRAPS tool – Synthesising Through Reflection And Participatory Sense-making) could provide a useful resource and starting point to other review teams who wish to evaluate co-production in their reviews and encourage others to share their experiences with us.&lt;/p&gt;\u0000 &lt;/section&gt;\u0000 \u0000 &lt;section&gt;\u0000 ","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"4 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2026-01-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12782252/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145954696","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Sensitivity Analysis in Meta-Analysis: A Tutorial","authors":"Nyan Min Aung,&nbsp;Ivan Jurak,&nbsp;Seemab Mehmood,&nbsp;Emma Axon","doi":"10.1002/cesm.70067","DOIUrl":"https://doi.org/10.1002/cesm.70067","url":null,"abstract":"<p>This tutorial explains when systematic review authors may consider performing a sensitivity analysis in a meta-analysis. Such scenarios include removing studies at high risk of bias, exploring the effect of outliers and examining differences in study characteristics (e.g., participants’ age, study design). In addition, examples are provided, as well as advice on how to interpret and report the results. The tutorial also explains the differences between subgroup and sensitivity analyses, as well as describing the disadvantages of a sensitivity analysis. To support this tutorial, a link to an online module, which includes videos and quizzes, is also provided.</p>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"4 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2026-01-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70067","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145909311","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Responsible Integration of Artificial Intelligence in Rapid Reviews: A Position Statement From the Cochrane Rapid Reviews Methods Group","authors":"Gerald Gartlehner,&nbsp;Barbara Nussbaumer-Streit,&nbsp;Candyce Hamel,&nbsp;Chantelle Garritty,&nbsp;Ursula Griebler,&nbsp;Valerie Jean King,&nbsp;Declan Devane,&nbsp;Chris Kamel","doi":"10.1002/cesm.70063","DOIUrl":"https://doi.org/10.1002/cesm.70063","url":null,"abstract":"&lt;p&gt;Rapidly evolving artificial intelligence (AI) technologies are increasingly used to accelerate literature review processes. A recent review and evidence map identified almost 100 studies published since 2021assessing AI applications in evidence synthesis [&lt;span&gt;1&lt;/span&gt;]. These technologies span from machine-learning classifiers to generative large-language models (LLMs). Recently, a preprint reported that a tool powered by LLMs autonomously reproduced and updated 12 Cochrane reviews in just 2 days [&lt;span&gt;2&lt;/span&gt;], sparking debate about when and how AI can be used safely and effectively to support systematic and rapid reviews.&lt;/p&gt;&lt;p&gt;In this position statement, the Cochrane Rapid Reviews Methods Group outlines its stance on the use of AI in rapid reviews. Rapid reviews encompass various types of evidence synthesis, and while some AI tools have been developed for specific review types, such as qualitative evidence syntheses, most are designed for more general application across review methodologies.&lt;/p&gt;&lt;p&gt;The main recommendations are summarized in Textbox 1. They complement a recently released position statement by Cochrane and other evidence synthesis organizations on the use of AI in evidence synthesis [&lt;span&gt;3&lt;/span&gt;].&lt;/p&gt;&lt;p&gt;Semi-automation of discrete steps in the evidence synthesis process —where algorithms assist but do not replace human reviewers—is not new. Cochrane, for instance, was an early adopter with the development of the randomized controlled trial (RCT) Classifier, a machine learning tool that identifies RCTs during abstract screening [&lt;span&gt;4&lt;/span&gt;]. Semi-automation plays a different role in rapid reviews than in traditional systematic reviews, where methodological certainty is typically prioritized. Because rapid reviews already balance rigor and timeliness, teams may be more willing to adopt efficiency-enhancing tools sooner.&lt;/p&gt;&lt;p&gt;The advent of generative LLMs, such as ChatGPT [&lt;span&gt;5&lt;/span&gt;] or Gemini [&lt;span&gt;6&lt;/span&gt;], has substantially expanded the potential for AI to support tasks in evidence synthesis. Unlike earlier machine learning tools that required extensive task-specific training data, LLMs can be deployed in zero-shot settings—meaning they can be applied without prior training or fine-tuning to a given task. This dramatically lowers the barrier to entry, offering a more accessible pathway for integrating AI into review workflows. Multiple studies have assessed the utility of generative LLMs to support the development of search strategies [&lt;span&gt;7&lt;/span&gt;], literature screening [&lt;span&gt;8-10&lt;/span&gt;], risk of bias assessment [&lt;span&gt;11, 12&lt;/span&gt;], and data extraction [&lt;span&gt;8, 13-15&lt;/span&gt;]. However, findings to date indicate highly variable performance ranging from high accuracy in some tasks to concerning errors in others [&lt;span&gt;1&lt;/span&gt;]. In parallel, developers of literature review software have begun integrating LLMs into their products.&lt;/p&gt;&lt;p&gt;Importantly, in rapid reviews, AI has the potential not only to enha","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-11-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70063","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145625659","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Introducing a Series of Reviews Assessing Engagement in Evidence Syntheses","authors":"Jennifer Petkovic,&nbsp;Joanne Khabsa,&nbsp;Lyubov Lytvyn,&nbsp;Alex Todhunter-Brown,&nbsp;Olivia Magwood,&nbsp;Pauline Campbell,&nbsp;Elie A. Akl,&nbsp;Thomas W. Concannon,&nbsp;Holger Schunemann,&nbsp;Vivian Welch,&nbsp;Peter Tugwell","doi":"10.1002/cesm.70057","DOIUrl":"https://doi.org/10.1002/cesm.70057","url":null,"abstract":"&lt;p&gt;High quality evidence syntheses are used in health decision-making, such as policies, legislation, and clinical recommendations [&lt;span&gt;1&lt;/span&gt;]. The usefulness, relevance, meaningfulness, and accessibility of evidence syntheses may be improved when people who are affected by those decisions, called “interest-holders,” are included in the evidence synthesis process [&lt;span&gt;2-4&lt;/span&gt;]. This concept of engagement in research is based on the principle that those affected by the health condition under study or the intervention to address it have a moral right to contribute to the decisions about how the research is conducted [&lt;span&gt;3, 5&lt;/span&gt;]. While there are increasing expectations from funders regarding the involvement of interest-holders [&lt;span&gt;6&lt;/span&gt;], the most effective methods for engaging different interest-holders in evidence syntheses have not been identified [&lt;span&gt;5&lt;/span&gt;]. Additionally, while there is some guidance related to engagement in research, it predominantly focuses on patient and public engagement in primary research, not evidence synthesis and there is limited guidance for engaging with other interest-holders [&lt;span&gt;3, 4, 7-9&lt;/span&gt;].&lt;/p&gt;&lt;p&gt;The aim of this paper is to introduce a series of articles about how to successfully engage different interest-holders when conducting evidence syntheses. The series of articles will consider methods used to engage different interest-holders (including who to involve and in what way), barriers and facilitators to engagement, impacts of engagement, management of conflicts of interest, and factors relating to equity.&lt;/p&gt;&lt;p&gt;This paper presents the shared definitions used across each of the five reviews included in this series. These reviews will inform the development of a guidance checklist and resources for engaging interest-holders through all steps of evidence synthesis. The plan for developing this guidance is described in the project protocol [&lt;span&gt;10&lt;/span&gt;].&lt;/p&gt;&lt;p&gt;“Interest-holders” are groups of people with legitimate interests in the health issue under consideration and whose perspectives and views should be considered when conducting this study [&lt;span&gt;2&lt;/span&gt;]. Their interests arise and draw their legitimacy from the fact that these people are responsible for or affected by health- and healthcare-related decisions that can be informed by research evidence. Engagement of interest-holders in evidence syntheses can promote transparency, accountability, trust, and help to ensure that the needs of interest-holders are included. Engagement can improve the translation of evidence into policy and practice [&lt;span&gt;11&lt;/span&gt;]. Interest-holders can contribute throughout the steps of evidence synthesis including, for example, refining the research question and suggesting appropriate outcomes, suggesting additional references to consider, and providing context to interpret the evidence.&lt;/p&gt;&lt;p&gt;This study was conducted by the MuSE Consortium, a group of over 160 individuals from 20 countrie","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-11-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70057","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145581023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Assessing the Feasibility and Acceptability of a Bespoke Large Language Model Pipeline to Extract Data From Different Study Designs for Public Health Evidence Reviews","authors":"Zalaya Simmons,&nbsp;Beti Evans,&nbsp;Tamsyn Harris,&nbsp;Harry Woolnough,&nbsp;Lauren Dunn,&nbsp;Jonathon Fuller,&nbsp;Kerry Cella,&nbsp;Daphne Duval","doi":"10.1002/cesm.70061","DOIUrl":"10.1002/cesm.70061","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>Data extraction is a critical but resource-intensive step of the evidence review process. Whilst there is evidence that artificial intelligence (AI) and large language models (LLMs) can improve the efficiency of data extraction from randomized controlled trials, their potential for other study designs is unclear. In this context, this study aimed to evaluate the performance of a bespoke LLM model pipeline (Retrieval-Augmented Generation pipeline utilizing LLaMa 3-70B) to automate data extraction from a range of study designs by assessing the accuracy and reliability of the extractions measured as error types and acceptability.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>Accuracy was assessed by retrospectively comparing the LLM extractions against human extractions from a review previously conducted by the authors. A total of 173 data fields from 24 articles (including experimental, observational, qualitative, and modeling studies) were assessed, of which three were used for prompt engineering. Reliability was assessed by calculating the mean maximum agreement rate (the highest proportion of identical returns from 10 consecutive extractions) for 116 data fields from 16 of the 24 studies. An evaluation framework was developed to assess the accuracy and reliability of LLM outputs measured as error types and acceptability (acceptability was assessed on whether it would be usable in real-world settings if the model acted as one reviewer and a human as a second reviewer).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Of the 173 data fields evaluated for accuracy, 68% were rated by human reviewers as acceptable (consistent with what is deemed to be acceptable data extraction from a human reviewer). However, acceptability ratings varied depending on the data field extracted (33% to 100%), with at least 90% acceptability for “objective,” “setting,” and “study design,” but 54% or less for data fields such as “outcome” and “time period.” For reliability, the mean maximum agreement rate was 0.71 (SD: 0.28), with variation across different data fields.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>This evaluation demonstrates the potential for LLMs, when paired with human quality assurance, to support data extraction in evidence reviews that include a range of study designs. However, further improvements in performance and validation are required before the model can be introduced into review workflows.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12584109/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145454519","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Health Equity in Systematic Reviews: A Tutorial—Part 1 Getting Started With Health Equity in Your Review","authors":"Jennifer Petkovic,&nbsp;Jordi Pardo Pardo,&nbsp;Vivian Welch,&nbsp;Omar Dewidar,&nbsp;Lara J. Maxwell,&nbsp;Andrea Darzi,&nbsp;Tamara Lotfi,&nbsp;Lawrence Mbuagbaw,&nbsp;Kevin Pottie,&nbsp;Peter Tugwell","doi":"10.1002/cesm.70055","DOIUrl":"https://doi.org/10.1002/cesm.70055","url":null,"abstract":"<p>This tutorial focuses on how to get started with considering health equity in systematic reviews of interventions. We will explain why health equity should be considered, how to frame your question, and which interest-holders to engage. This is the first tutorial in a series on health equity. The second tutorial focuses on implementing health equity methods in your review.</p>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70055","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145406969","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Harnessing Large-Language Models for Efficient Data Extraction in Systematic Reviews: The Role of Prompt Engineering","authors":"Molly Murton,&nbsp;Ellie Boulton,&nbsp;Shona Cross,&nbsp;Ambar Khan,&nbsp;Swati Kumar,&nbsp;Giuseppina Magri,&nbsp;Charlotte Marris,&nbsp;David Slater,&nbsp;Emma Worthington,&nbsp;Elizabeth Lunn","doi":"10.1002/cesm.70058","DOIUrl":"10.1002/cesm.70058","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>Systematic literature reviews (SLRs) of randomized clinical trials (RCTs) underpin evidence-based medicine but can be limited by the intensive resource demands of data extraction. Recent advances in accessible large-language models (LLMs) hold promise for automating this step, however testing is limited across different outcomes and disease areas.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>This study developed prompt engineering strategies for GPT-4o to extract data from RCTs across three disease areas: non-small cell lung cancer, endometrial cancer and hypertrophic cardiomyopathy. Prompts were iteratively refined during the development phase, then tested on unseen data. Performance was evaluated via comparison to human extraction of the same data, using F1 scores, precision, recall and percentage accuracy.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The LLM was highly effective for extracting study and baseline characteristics, often equaling human performance, with test F1 scores exceeding 0.85. Complex efficacy and adverse event data proved more challenging, with test F1 scores ranging from 0.22 to 0.50. Transferability of prompts across disease areas was promising but varied, highlighting the need for disease-specific refinement.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>Our findings demonstrate the potential of LLMs, guided by rigorous prompt engineering, to augment the SLR process. However, human oversight remains essential, particularly for complex and nuanced data. As these technologies evolve, continued validation of AI tools will be necessary to ensure accuracy and reliability, and safeguarding of the quality of evidence synthesis.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12559671/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145403637","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Human-in-the-Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software","authors":"Kevin M. Kallmes,&nbsp;Jade Thurnham,&nbsp;Marius Sauca,&nbsp;Ranita Tarchand,&nbsp;Keith R. Kallmes,&nbsp;Karl J. Holub","doi":"10.1002/cesm.70059","DOIUrl":"https://doi.org/10.1002/cesm.70059","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Introduction</h3>\u0000 \u0000 <p>While artificial intelligence (AI) tools have been utilized for individual stages within the systematic literature review (SLR) process, no tool has previously been shown to support each critical SLR step. In addition, the need for expert oversight has been recognized to ensure the quality of SLR findings. Here, we describe a complete methodology for utilizing our AI SLR tool with human-in-the-loop curation workflows, as well as AI validations, time savings, and approaches to ensure compliance with best review practices.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>SLRs require completing Search, Screening, and Extraction from relevant studies, with meta-analysis and critical appraisal as relevant. We present a full methodological framework for completing SLRs utilizing our AutoLit software (Nested Knowledge). This system integrates AI models into the central steps in SLR: Search strategy generation, Dual Screening of Titles/Abstracts and Full Texts, and Extraction of qualitative and quantitative evidence. The system also offers manual Critical Appraisal and Insight drafting and fully-automated Network Meta-analysis. Validations comparing AI performance to experts are reported, and where relevant, time savings and ‘rapid review’ alternatives to the SLR workflow.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Search strategy generation with the Smart Search AI can turn a Research Question into full Boolean strings with 76.8% and 79.6% Recall in two validation sets. Supervised machine learning tools can achieve 82–97% Recall in reviewer-level Screening. Population, Interventions/Comparators, and Outcomes (PICOs) extraction achieved F1 of 0.74; accuracy for study type, location, and size were 74%, 78%, and 91%, respectively. Time savings of 50% in Abstract Screening and 70–80% in qualitative extraction were reported. Extraction of user-specified qualitative and quantitative tags and data elements remains exploratory and requires human curation for SLRs.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>AI systems can support high-quality, human-in-the-loop execution of key SLR stages. Transparency, replicability, and expert oversight are central to the use of AI SLR tools.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/cesm.70059","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145367149","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Enhancing Evidence Synthesis Efficiency: Leveraging Large Language Models and Agentic Workflows for Optimized Literature Screening","authors":"Bing Hu,&nbsp;Emmalie Tomini,&nbsp;Tricia Corrin,&nbsp;Kusala Pussegoda,&nbsp;Elias Sandner,&nbsp;Andre Henriques,&nbsp;Alice Simniceanu,&nbsp;Luca Fontana,&nbsp;Andreas Wagner,&nbsp;Stephanie Brazeau,&nbsp;Lisa Waddell","doi":"10.1002/cesm.70042","DOIUrl":"10.1002/cesm.70042","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Public health events of international concern highlight the need for up-to-date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic-AI assisted pipeline (GREP-Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>In this study, the performance of the GREP-Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o-mini, Llama3.1, and Phi4). The GREP-Agent framework integrates multiple LLMs and human feedback to fine-tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic-AI system is presented and measured by accuracy, precision, recall, and F1-score metrics.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The results show how each phase of the GREP-Agent system incrementally improves accuracy regardless of the LLM. We found that GREP-Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%–88.9% after fine-tuning and to 86.4%–95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>The GREP-Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human-in-the-loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human-centric approach.</p>\u0000 </section>\u0000 </div>","PeriodicalId":100286,"journal":{"name":"Cochrane Evidence Synthesis and Methods","volume":"3 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12538819/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145351061","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}