Cochrane Evidence Synthesis and Methods最新文献

Objective

PubReMiner is a text-mining tool that analyses a seed set of citations to assess word frequency in titles, abstracts, and Medical Subject Headings (MeSH). This study aimed to determine the sensitivity and precision of search strategies developed using the PubReMiner tool compared to conventional search strategies developed by a librarian at our institution.

Methods

Twelve consecutive reviews conducted at our center were included from September 2023 to January 2025. These reviews included various types of evidence synthesis, including rapid reviews and systematic reviews, covering a variety of topics. One librarian developed a comprehensive search strategy, which included a conventional MEDLINE search for each review. Separately, two librarians independently developed MEDLINE search strategies using PubReMiner-generated word frequency tables (PubReMiner 1 and PubReMiner 2). All search strategies were constructed by experienced librarians using predefined work instructions. Primary outcomes were sensitivity and precision. Secondary outcomes included the number needed to read, the number of unique references retrieved, and the time taken to construct each strategy.

Results

Sensitivity of PubReMiner strategies was generally lower than that of conventional strategies; however, in one review, PubReMiner achieved a higher sensitivity (83.87%) than the conventional strategy (58.06%). Only the sensitivity outcome showed a statistically significant difference between search methods (Friedman test p = 0.0065). No statistically significant difference in precision between the searches was identified. PubReMiner strategies were typically faster to construct but yielded inconsistent performance across reviews and between librarians.

Conclusion

While PubReMiner offers efficiency advantages, its inconsistent performance in retrieving relevant studies suggests that it should not replace conventional search strategies. The study illustrates the value of multi-review SWARs in producing evidence that informs evidence synthesis practices.

Introduction

Co-production of evidence syntheses has the potential to facilitate translation of research findings into policy and practice. However, few studies have explored the process of implementing co-produced evidence. This gap limits our understanding of how, and to what extent, co-production promotes knowledge translation.

In this study, we used an implementation science lens to explore factors influencing the implementation of the Best Practice Principles in partnering with consumers (BPP) in hospitals in Melbourne, Australia. The BPP were developed as part of a co-produced Cochrane qualitative evidence synthesis exploring consumers' and health providers' experiences and perceptions of partnering. We use the findings of our study to develop strategies for evidence synthesis teams engaged in co-production to optimize the implementation of their review findings.

Methods

This exploratory, qualitative study was informed by cooperative inquiry and normalization process theory (NPT). A six-member panel, including researchers, policy makers and consumers, guided data collection and analysis. Data collection involved semi-structured interviews with eleven participants (including consumer engagement leads, consumer representatives, and a policymaker) about how to implement the BPP in Melbourne hospitals. Interviews were analyzed using framework analysis.

Results

Interview participants reported the BPP were relevant to practice, consumer-centered, practical, and flexible. There were several additional factors that could impact their uptake into practice. These included integration of the BPP into government policies and guidelines, evidence of the cost/benefit of BPP implementation, endorsement from health service leadership, involvement of consumers throughout the implementation process, a structured implementation, and flexible measurement of implementation success.

Conclusion

This exploratory study suggested that the BPP, a tool developed through a co-produced Cochrane qualitative evidence synthesis, promoted knowledge translation. Other factors at the macro- (political and economic), meso- (systems and organizations), and micro- (individual) levels could influence the implementation's success. Implications for evidence synthesis teams aiming to optimize the knowledge translation of their review results are discussed.

Introduction

Priority screening has the potential to reduce the number of records that need to be annotated in systematic literature reviews. So-called technology-assisted reviews (TAR) use machine-learning with prior include/exclude annotations to continuously rank unseen records by their predicted relevance to find relevant records earlier. In this article, we present a systematic evaluation of methods to determine when it is safe to stop screening when using prioritization.

Methods

We implement an open-source evaluation framework that features a novel method to generate rankings and simulate priority screening processes for 81 real-world data sets. We use these simulations to evaluate 15 statistical or rule-based (heuristic) stopping methods, testing a range of hyperparameters for each.

Results

The work-saving potential and performance of stopping criteria heavily rely on “good” rankings, which are typically not achieved by a single ranking algorithm across the entire screening process. Our evaluation shows that almost all existing stopping methods either fail to reliably stop without missing relevant records or fail to utilize the full potential work-savings. Only one method reliably meets the set recall target, but stops conservatively.

Conclusions

Many digital evidence synthesis tools provide priority screening features that are already used in many research projects. However, the theoretical work-savings demonstrated in retrospective simulations of prioritization can only be unlocked with safe and reproducible stopping criteria. Our results highlight the need for improved stopping methods and guidelines on how to responsibly use priority screening. We also urge screening platforms to provide indicators and authors to transparently report metrics when automating (parts of) their synthesis.

Background

Almost a decade ago, about half of biomedical PhD programs across Europe specifically stated that systematic reviews could not be accepted as part of a doctoral thesis, illustrating limited merit value at that time. The aim of this study was to explore current Swedish university policies on this research design.

Methods

Policy documents for PhD theses and applications to associate professor positions were obtained from all medical faculties at universities in Sweden. Instructions regarding systematic reviews, with focus on their merit value and related aspects, were independently extracted and categorized by two authors, with discrepancies resolved in consensus discussions.

Results

All seven medical faculties accepted at least one systematic review within a PhD thesis, five restricted the number of such studies accepted, and five provided instructions regarding this study design. Regarding policies for promotion to associate professor, six medical faculties accepted at least one published systematic review to merit recognition―the remaining one required meta-analyses for acceptance―and three explicitly restricted the number of systematic reviews. No restrictions or guidance were provided for other designs intended to answer specific research questions.

Conclusion

As of 2025, systematic reviews appear to be generally recognized as contributing to authors' academic merit. For this research design exclusively, some universities impose restrictions that may limit their recognition, and some provide guidance which may help ensure quality in reporting. These findings may encourage research to evaluate the merit value of systematic reviews in other settings, and to examine potential implications of restrictions and guidance in policy documents.