Research Synthesis Methods最新文献

Objective: This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews.

Study design and setting: The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group.

Results: This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de-duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability.

Conclusion: This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high-quality scoping reviews.

The application of network meta-analysis is becoming increasingly widespread, and for a successful implementation, it requires that the direct comparison result and the indirect comparison result should be consistent. Because of this, a proper detection of inconsistency is often a key issue in network meta-analysis as whether the results can be reliably used as a clinical guidance. Among the existing methods for detecting inconsistency, two commonly used models are the design-by-treatment interaction model and the side-splitting models. While the original side-splitting model was initially estimated using a Bayesian approach, in this context, we employ the frequentist approach. In this paper, we review these two types of models comprehensively as well as explore their relationship by treating the data structure of network meta-analysis as missing data and parameterizing the potential complete data for each model. Through both analytical and numerical studies, we verify that the side-splitting models are specific instances of the design-by-treatment interaction model, incorporating additional assumptions or under certain data structure. Moreover, the design-by-treatment interaction model exhibits robust performance across different data structures on inconsistency detection compared to the side-splitting models. Finally, as a practical guidance for inconsistency detection, we recommend utilizing the design-by-treatment interaction model when there is a lack of information about the potential location of inconsistency. By contrast, the side-splitting models can serve as a supplementary method especially when the number of studies in each design is small, enabling a comprehensive assessment of inconsistency from both global and local perspectives.

The development of new statistical models for the meta-analysis of diagnostic test accuracy studies is still an ongoing field of research, especially with respect to summary receiver operating characteristic (ROC) curves. In the recently published updated version of the "Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy", the authors point to the challenges of this kind of meta-analysis and propose two approaches. However, both of them come with some disadvantages, such as the nonstraightforward choice of priors in Bayesian models or the requirement of a two-step approach where parameters are estimated for the individual studies, followed by summarizing the results. As an alternative, we propose a novel model by applying methods from time-to-event analysis. To this task we use the discrete proportional hazard approach to treat the different diagnostic thresholds, that provide means to estimate sensitivity and specificity and are reported by the single studies, as categorical variables in a generalized linear mixed model, using both the logit- and the asymmetric cloglog-link. This leads to a model specification with threshold-specific discrete hazards, avoiding a linear dependency between thresholds, discrete hazard, and sensitivity/specificity and thus increasing model flexibility. We compare the resulting models to approaches from the literature in a simulation study. While the estimated area under the summary ROC curve is estimated comparably well in most approaches, the results depict substantial differences in the estimated sensitivities and specificities. We also show the practical applicability of the models to data from a meta-analysis for the screening of type 2 diabetes.

Multi-lab projects are large scale collaborations between participating data collection sites that gather empirical evidence and (usually) analyze that evidence using meta-analyses. They are a valuable form of scientific collaboration, produce outstanding data sets and are a great resource for third-party researchers. Their data may be reanalyzed and used in research synthesis. Their repositories and code could provide guidance to future projects of this kind. But, while multi-labs are similar in their structure and aggregate their data using meta-analyses, they deploy a variety of different solutions regarding the storage structure in the repositories, the way the (analysis) code is structured and the file-formats they provide. Continuing this trend implies that anyone who wants to work with data from multiple of these projects, or combine their datasets, is faced with an ever-increasing complexity. Some of that complexity could be avoided. Here, we introduce MetaPipeX, a standardized framework to harmonize, document and analyze multi-lab data. It features a pipeline conceptualization of the analysis and documentation process, an R-package that implements both and a Shiny App (https://www.apps.meta-rep.lmu.de/metapipex/) that allows users to explore and visualize these data sets. We introduce the framework by describing its components and applying it to a practical example. Engaging with this form of collaboration and integrating it further into research practice will certainly be beneficial to quantitative sciences and we hope the framework provides a structure and tools to reduce effort for anyone who creates, re-uses, harmonizes or learns about multi-lab replication projects.

In sparse data meta-analyses (with few trials or zero events), conventional methods may distort results. Although better-performing one-stage methods have become available in recent years, their implementation remains limited in practice. This study examines the impact of using conventional methods compared to one-stage models by re-analysing meta-analyses from the Cochrane Database of Systematic Reviews in scenarios with zero event trials and few trials. For each scenario, we computed one-stage methods (Generalised linear mixed model [GLMM], Beta-binomial model [BBM], Bayesian binomial-normal hierarchical model using a weakly informative prior [BNHM-WIP]) and compared them with conventional methods (Peto-Odds-ratio [PETO], DerSimonian-Laird method [DL] for zero event trials; DL, Paule-Mandel [PM], Restricted maximum likelihood [REML] method for few trials). While all methods showed similar treatment effect estimates, substantial variability in statistical precision emerged. Conventional methods generally resulted in smaller confidence intervals (CIs) compared to one-stage models in the zero event situation. In the few trials scenario, the CI lengths were widest for the BBM on average and significance often changed compared to the PM and REML, despite the relatively wide CIs of the latter. In agreement with simulations and guidelines for meta-analyses with zero event trials, our results suggest that one-stage models are preferable. The best model can be either selected based on the data situation or, using a method that can be used in various situations. In the few trial situation, using BBM and additionally PM or REML for sensitivity analyses appears reasonable when conservative results are desired. Overall, our results encourage careful method selection.

Conducting high-quality overviews of reviews (OoR) is time-consuming. Because the quality of systematic reviews (SRs) varies, it is necessary to critically appraise SRs when conducting an OoR. A well-established appraisal tool is A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2, which takes about 15-32 min per application. To save time, we developed two fast-and-frugal decision trees (FFTs) for assessing the methodological quality of SR for OoR either during the full-text screening stage (Screening FFT) or to the resulting pool of SRs (Rapid Appraisal FFT). To build a data set for developing the FFT, we identified published AMSTAR 2 appraisals. Overall confidence ratings of the AMSTAR 2 were used as a criterion and the 16 items as cues. One thousand five hundred and nineteen appraisals were obtained from 24 publications and divided into training and test data sets. The resulting Screening FFT consists of three items and correctly identifies all non-critically low-quality SRs (sensitivity of 100%), but has a positive predictive value of 59%. The three-item Rapid Appraisal FFT correctly identifies 80% of the high-quality SRs and correctly identifies 97% of the low-quality SRs, resulting in an accuracy of 95%. The FFTs require about 10% of the 16 AMSTAR 2 items. The Screening FFT may be applied during full-text screening to exclude SRs with critically low quality. The Rapid Appraisal FFT may be applied to the final SR pool to identify SR that might be of high methodological quality.

In response to the evolving needs of knowledge synthesis, this manuscript introduces the concept of narrative reanalysis, a method that refines data from initial reviews, such as systematic and reviews, to focus on specific sub-phenomena. Unlike traditional narrative reviews, which lack the methodological rigor of systematic reviews and are broader in scope, our methodological framework for narrative reanalysis applies a structured, systematic framework to the interpretation of existing data. This approach enables a focused investigation of nuanced topics within a broader dataset, enhancing understanding and generating new insights. We detail a five-stage methodological framework that guides the narrative reanalysis process: (1) retrieval of an initial review, (2) identification and justification of a sub-phenomenon, (3) expanded search, selection, and extraction of data, (4) reanalyzing the sub-phenomenon, and (5) writing the report. The proposed framework aims to standardize narrative reanalysis, advocating for its use in academic and research settings to foster more rigorous and insightful literature reviews. This approach bridges the methodological gap between narrative and systematic reviews, offering a valuable tool for researchers to explore detailed aspects of broader topics without the extensive resources required for systematic reviews.

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two different approaches: (1) manually by human reviewers, and (2) automatically by the RobotReviewer. The manual assessment was based on two groups independently, with two additional rounds of verification. The agreement between RobotReviewer and humans was measured via the concordance rate and Cohen's kappa statistics, based on the comparison of binary classification of the risk of bias (low vs. high/unclear) as restricted by RobotReviewer. The concordance rates varied by domain, ranging from 63.07% to 83.32%. Cohen's kappa statistics showed a poor agreement between humans and RobotReviewer for allocation concealment (κ = 0.25, 95% CI: 0.21-0.30), blinding of outcome assessors (κ = 0.27, 95% CI: 0.23-0.31); While moderate for random sequence generation (κ = 0.46, 95% CI: 0.41-0.50) and blinding of participants and personnel (κ = 0.59, 95% CI: 0.55-0.64). The findings demonstrate that there were domain-specific differences in the level of agreement between RobotReviewer and humans. We suggest that it might be a useful auxiliary tool, but the specific manner of its integration as a complementary tool requires further discussion.

Researchers may have at their disposal the raw data of the studies they wish to meta-analyze. The goal of this study is to identify, illustrate, and compare a range of possible analysis options for researchers to whom raw data are available, wanting to fit a structural equation model (SEM) to these data. This study illustrates techniques that directly analyze the raw data, such as multilevel and multigroup SEM, and techniques based on summary statistics, such as correlation-based meta-analytical structural equation modeling (MASEM), discussing differences in procedures, capabilities, and outcomes. This is done by analyzing a previously published collection of datasets using open source software. A path model reflecting the theory of planned behavior is fitted to these datasets using different techniques involving SEM. Apart from differences in handling of missing data, the ability to include study-level moderators, and conceptualization of heterogeneity, results show differences in parameter estimates and standard errors across methods. Further research is needed to properly formulate guidelines for applied researchers looking to conduct individual participant data MASEM.

Increasing integrity concerns in medical research have prompted the development of tools to detect untrustworthy studies. Existing tools primarily assess published aggregate data (AD), though scrutiny of individual participant data (IPD) is often required to detect trustworthiness issues. Thus, we developed the IPD Integrity Tool for detecting integrity issues in randomised trials with IPD available. This manuscript describes the development of this tool. We conducted a literature review to collate and map existing integrity items. These were discussed with an expert advisory group; agreed items were included in a standardised tool and automated where possible. We piloted this tool in two IPD meta-analyses (including 116 trials) and conducted preliminary validation checks on 13 datasets with and without known integrity issues. We identified 120 integrity items: 54 could be conducted using AD, 48 required IPD, and 18 were possible with AD, but more comprehensive with IPD. An initial reduced tool was developed through consensus involving 13 advisors, featuring 11 AD items across four domains, and 12 IPD items across eight domains. The tool was iteratively refined throughout piloting and validation. All studies with known integrity issues were accurately identified during validation. The final tool includes seven AD domains with 13 items and eight IPD domains with 18 items. The quality of evidence informing healthcare relies on trustworthy data. We describe the development of a tool to enable researchers, editors, and others to detect integrity issues using IPD. Detailed instructions for its application are published as a complementary manuscript in this issue.