Cochrane Evidence Synthesis and Methods最新文献

Introduction

Co-production of evidence syntheses has the potential to facilitate translation of research findings into policy and practice. However, few studies have explored the process of implementing co-produced evidence. This gap limits our understanding of how, and to what extent, co-production promotes knowledge translation.

In this study, we used an implementation science lens to explore factors influencing the implementation of the Best Practice Principles in partnering with consumers (BPP) in hospitals in Melbourne, Australia. The BPP were developed as part of a co-produced Cochrane qualitative evidence synthesis exploring consumers' and health providers' experiences and perceptions of partnering. We use the findings of our study to develop strategies for evidence synthesis teams engaged in co-production to optimize the implementation of their review findings.

Methods

This exploratory, qualitative study was informed by cooperative inquiry and normalization process theory (NPT). A six-member panel, including researchers, policy makers and consumers, guided data collection and analysis. Data collection involved semi-structured interviews with eleven participants (including consumer engagement leads, consumer representatives, and a policymaker) about how to implement the BPP in Melbourne hospitals. Interviews were analyzed using framework analysis.

Results

Interview participants reported the BPP were relevant to practice, consumer-centered, practical, and flexible. There were several additional factors that could impact their uptake into practice. These included integration of the BPP into government policies and guidelines, evidence of the cost/benefit of BPP implementation, endorsement from health service leadership, involvement of consumers throughout the implementation process, a structured implementation, and flexible measurement of implementation success.

Conclusion

This exploratory study suggested that the BPP, a tool developed through a co-produced Cochrane qualitative evidence synthesis, promoted knowledge translation. Other factors at the macro- (political and economic), meso- (systems and organizations), and micro- (individual) levels could influence the implementation's success. Implications for evidence synthesis teams aiming to optimize the knowledge translation of their review results are discussed.

Introduction

Priority screening has the potential to reduce the number of records that need to be annotated in systematic literature reviews. So-called technology-assisted reviews (TAR) use machine-learning with prior include/exclude annotations to continuously rank unseen records by their predicted relevance to find relevant records earlier. In this article, we present a systematic evaluation of methods to determine when it is safe to stop screening when using prioritization.

Methods

We implement an open-source evaluation framework that features a novel method to generate rankings and simulate priority screening processes for 81 real-world data sets. We use these simulations to evaluate 15 statistical or rule-based (heuristic) stopping methods, testing a range of hyperparameters for each.

Results

The work-saving potential and performance of stopping criteria heavily rely on “good” rankings, which are typically not achieved by a single ranking algorithm across the entire screening process. Our evaluation shows that almost all existing stopping methods either fail to reliably stop without missing relevant records or fail to utilize the full potential work-savings. Only one method reliably meets the set recall target, but stops conservatively.

Conclusions

Many digital evidence synthesis tools provide priority screening features that are already used in many research projects. However, the theoretical work-savings demonstrated in retrospective simulations of prioritization can only be unlocked with safe and reproducible stopping criteria. Our results highlight the need for improved stopping methods and guidelines on how to responsibly use priority screening. We also urge screening platforms to provide indicators and authors to transparently report metrics when automating (parts of) their synthesis.

Background

Almost a decade ago, about half of biomedical PhD programs across Europe specifically stated that systematic reviews could not be accepted as part of a doctoral thesis, illustrating limited merit value at that time. The aim of this study was to explore current Swedish university policies on this research design.

Methods

Policy documents for PhD theses and applications to associate professor positions were obtained from all medical faculties at universities in Sweden. Instructions regarding systematic reviews, with focus on their merit value and related aspects, were independently extracted and categorized by two authors, with discrepancies resolved in consensus discussions.

Results

All seven medical faculties accepted at least one systematic review within a PhD thesis, five restricted the number of such studies accepted, and five provided instructions regarding this study design. Regarding policies for promotion to associate professor, six medical faculties accepted at least one published systematic review to merit recognition―the remaining one required meta-analyses for acceptance―and three explicitly restricted the number of systematic reviews. No restrictions or guidance were provided for other designs intended to answer specific research questions.

Conclusion

As of 2025, systematic reviews appear to be generally recognized as contributing to authors' academic merit. For this research design exclusively, some universities impose restrictions that may limit their recognition, and some provide guidance which may help ensure quality in reporting. These findings may encourage research to evaluate the merit value of systematic reviews in other settings, and to examine potential implications of restrictions and guidance in policy documents.

Introduction

Data extraction is a critical but resource-intensive step of the evidence review process. Whilst there is evidence that artificial intelligence (AI) and large language models (LLMs) can improve the efficiency of data extraction from randomized controlled trials, their potential for other study designs is unclear. In this context, this study aimed to evaluate the performance of a bespoke LLM model pipeline (Retrieval-Augmented Generation pipeline utilizing LLaMa 3-70B) to automate data extraction from a range of study designs by assessing the accuracy and reliability of the extractions measured as error types and acceptability.

Methods

Accuracy was assessed by retrospectively comparing the LLM extractions against human extractions from a review previously conducted by the authors. A total of 173 data fields from 24 articles (including experimental, observational, qualitative, and modeling studies) were assessed, of which three were used for prompt engineering. Reliability was assessed by calculating the mean maximum agreement rate (the highest proportion of identical returns from 10 consecutive extractions) for 116 data fields from 16 of the 24 studies. An evaluation framework was developed to assess the accuracy and reliability of LLM outputs measured as error types and acceptability (acceptability was assessed on whether it would be usable in real-world settings if the model acted as one reviewer and a human as a second reviewer).

Results

Of the 173 data fields evaluated for accuracy, 68% were rated by human reviewers as acceptable (consistent with what is deemed to be acceptable data extraction from a human reviewer). However, acceptability ratings varied depending on the data field extracted (33% to 100%), with at least 90% acceptability for “objective,” “setting,” and “study design,” but 54% or less for data fields such as “outcome” and “time period.” For reliability, the mean maximum agreement rate was 0.71 (SD: 0.28), with variation across different data fields.

Conclusion

This evaluation demonstrates the potential for LLMs, when paired with human quality assurance, to support data extraction in evidence reviews that include a range of study designs. However, further improvements in performance and validation are required before the model can be introduced into review workflows.