Research Synthesis Methods最新文献

While the Institute of Education Science's ERIC is often recommended for comprehensive literature searching in the field of education, there are several other specialized education databases to discover education literature. This study investigates journal coverage overlaps between four specialized education databases: Education Source (EBSCO), Education Database (ProQuest), ERIC (Institute of Education Sciences), and Educator's Reference Complete (Gale). Out of a total of 4,695 unique journals analyzed, there are 2,831 journals uniquely covered by only one database, as well as many journals covered by only two or three databases. Findings show that evidence synthesis projects and literature reviews benefit from the careful selection of multiple specialized education databases and that ERIC is insufficient as the primary education database for comprehensive searching in the field.

The number of published systematic reviews has increased over the last years, with a non-negligible proportion displaying methodological concerns. We aimed to develop and evaluate a tool to assess the reported methodological quality of medical systematic reviews. The developed tool (ReMarQ) consists of 26 dichotomous items. We applied an item response theory model to assess the difficulty and discrimination of the items and decision tree models to identify those items more capable of identifying systematic reviews with higher reported methodological quality. ReMarQ was applied to a representative sample of medical systematic reviews (excluding those published in the Cochrane Database of Systematic Reviews) to describe their methodological quality and identify associated factors. We assessed 400 systematic reviews published between 2010 and 2020, of which 196 (49.0%) included meta-analysis. The most discriminative items were (i) conducting a risk of bias assessment, (ii) having a published protocol and (iii) reporting methods for solving disagreements. More recent systematic reviews (adjusted yearly RR=1.03; 95%CI=1.02 -1.04, p<0.001) and those with meta-analysis (adjusted RR=1.34; 95%CI=1.25 -1.43, p<0.001) were associated with higher reported methodological quality. Such an association was not observed with the journal impact factor. The items most frequently fulfilled were (i) reporting search dates, (ii) reporting bibliographic sources and (iii) searching multiple electronic bibliographic databases. ReMarQ, consisting of dichotomous items and whose application does not require subject content expertise, may be important (i) in supporting an efficient quality assessment of systematic reviews and (ii) as the basis of automated processes to support that assessment.

Data visualization is crucial for effectively communicating knowledge in meta-analysis. However, existing visualization methods in meta-analysis have predominantly focused on quantitative aspects, such as forest plots and funnel plots, thereby neglecting qualitative information that is equally important for end-users in science, policy, and practice. We introduce a framework consisting of a series of visualization toolkits designed to enrich meta-analyses by borrowing approaches from other research synthesis methods, including systematic evidence mapping (scoping reviews), bibliometrics (bibliometric analysis), and alternative impact metric analysis. These "enrichment" toolkits aim to facilitate the synthesis of both quantitative and qualitative evidence, along with the assessment of the academic and nonacademic influences of the meta-analytic evidence base. While the meta-analysis yields quantitative insights, the enrichment analyses, and visualizations provide user-friendly summaries of qualitative information on the evidence base. For example, a systematic evidence map can visualize study characteristics, unraveling knowledge gaps and methodological differences. Bibliometric analysis offers a visual assessment of the nonindependent evidence, such as hyper-dominant authors and countries, and funding sources, potentially informing the risk of bias. Alternative impact metric analysis employs alternative metrics to gauge societal influence and research translation (e.g., policy and patent citations) of studies in the meta-analysis. We provide a dedicated webpage showcasing sample visualizations and providing step-by-step implementation in open-source software R (https://yefeng0920.github.io/MA_Map_Bib/). Additionally, we offer a guide on leveraging three commercially free large language models (LLMs) to help adapt the sample script, enabling users with less R coding experience to visualize their own meta-analytic evidence base.

The product Bayes factor (PBF) synthesizes evidence for an informative hypothesis across heterogeneous replication studies. It can be used when fixed- or random effects meta-analysis fall short. For example, when effect sizes are incomparable and cannot be pooled, or when studies diverge significantly in the populations, study designs, and measures used. PBF shines as a solution for small sample meta-analyses, where the number of between-study differences is often large relative to the number of studies, precluding the use of meta-regression to account for these differences. Users should be mindful of the fact that the PBF answers a qualitatively different research question than other evidence synthesis methods. For example, whereas fixed-effect meta-analysis estimates the size of a population effect, the PBF quantifies to what extent an informative hypothesis is supported in all included studies. This tutorial paper showcases the user-friendly PBF functionality within the bain R-package. This new implementation of an existing method was validated using a simulation study, available in an Online Supplement. Results showed that PBF had a high overall accuracy, due to greater sensitivity and lower specificity, compared to random-effects meta-analysis, individual participant data meta-analysis, and vote counting. Tutorials demonstrate applications of the method on meta-analytic and individual participant data. The example datasets, based on published research, are included in bain so readers can reproduce the examples and apply the code to their own data. The PBF is a promising method for synthesizing evidence for informative hypotheses across conceptual replications that are not suitable for conventional meta-analysis.

Psychological interventions are complex in nature and have been shown to benefit various clinical outcomes. Gaining insight into current practices would help identify specific aspects that need improvement to enhance the quality of network meta-analysis (NMA) in this field. This scoping review aimed to explore methodological approaches in the NMA of psychological interventions. We searched PubMed, EMBASE, and Cochrane CENTRAL in September 2023. We included NMAs of psychological interventions of randomized controlled trials that reported clinical outcomes. Three independent researchers assessed the eligibility and extracted relevant data. The findings were presented using descriptive statistics. Of the 1827 articles identified, 187 studies were included. Prior protocol registration was reported in 130 studies (69.5%). Forty-six studies (24.6%) attempted to search for gray literature. Ninety-four studies (50.3%) explicitly assessed transitivity. Nearly three-quarters (143 studies, 76.5%) classified treatment nodes by the type of psychological intervention, while 13 studies (7.0%) did so by lumping different intervention types into more broader intervention classes. Seven studies (3.7%) examined active components of the intervention using component NMA. Only three studies (1.6%) classified interventions based on factors affecting intervention practices, specifically intensity, provider, and delivery platform. Meanwhile, 29 studies (15.5%) explored the influential effects of these factors using meta-regression, subgroup analysis, or sensitivity analysis. The certainty of evidence was assessed in 80 studies (42.8%). The methodological approach in NMAs of psychological interventions should be improved, specifically in classifying psychological interventions into treatment nodes, exploring the effects of intervention-related factors, and assessing the certainty of evidence.

Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although numerous stopping rules have been proposed to guide users in this decision, they have yet to undergo comprehensive evaluation. In this study, we evaluated the performance of three stopping rules: the knee method, a data-driven heuristic, and a prevalence estimation technique. We measured performance via sensitivity, specificity, and screening cost and explored the influence of the prevalence of relevant studies and the choice of the learning algorithm. We curated a dataset of abstract collections from meta-analyses across five psychological research domains. Our findings revealed performance differences between stopping rules regarding all performance measures and variations in the performance of stopping rules across different prevalence ratios. Moreover, despite the relatively minor impact of the learning algorithm, we found that specific combinations of stopping rules and learning algorithms were most effective for certain prevalence ratios of relevant abstracts. Based on these results, we derived practical recommendations for users of AI-aided screening tools. Furthermore, we discuss possible implications and offer suggestions for future research.

While geographic search filters exist, few of them are validated and there are currently none that focus on Germany. We aimed to develop and validate a highly sensitive geographic search filter for MEDLINE (PubMed) that identifies studies about Germany. First, using the relative recall method, we created a gold standard set of studies about Germany, dividing it into ‘development’ and ‘testing’ sets. Next, candidate search terms were identified using (i) term frequency analyses in the ‘development set’ and a random set of MEDLINE records; and (ii) a list of German geographic locations, compiled by our team. Then, we iteratively created the filter, evaluating it against the ‘development’ and ‘testing’ sets. To validate the filter, we conducted a number of case studies (CSs) and a simulation study. For this validation we used systematic reviews (SRs) that had included studies about Germany but did not restrict their search strategy geographically. When applying the filter to the original search strategies of the 17 SRs eligible for CSs, the median precision was 2.64% (interquartile range [IQR]: 1.34%–6.88%) versus 0.16% (IQR: 0.10%–0.49%) without the filter. The median number-needed-to-read (NNR) decreased from 625 (IQR: 211–1042) to 38 (IQR: 15–76). The filter achieved 100% sensitivity in 13 CSs, 85.71% in 2 CSs and 87.50% and 80% in the remaining 2 CSs. In a simulation study, the filter demonstrated an overall sensitivity of 97.19% and NNR of 42. The filter reliably identifies studies about Germany, enhancing screening efficiency and can be applied in evidence syntheses focusing on Germany.

Quantitative evidence synthesis methods aim to combine data from multiple medical trials to infer relative effects of different interventions. A challenge arises when trials report continuous outcomes on different measurement scales. To include all evidence in one coherent analysis, we require methods to “map” the outcomes onto a single scale. This is particularly challenging when trials report aggregate rather than individual data. We are motivated by a meta-analysis of interventions to prevent obesity in children. Trials report aggregate measurements of body mass index (BMI) either expressed as raw values or standardized for age and sex. We develop three methods for mapping between aggregate BMI data using known or estimated relationships between measurements on different scales at the individual level. The first is an analytical method based on the mathematical definitions of z-scores and percentiles. The other two approaches involve sampling individual participant data on which to perform the conversions. One method is a straightforward sampling routine, while the other involves optimization with respect to the reported outcomes. In contrast to the analytical approach, these methods also have wider applicability for mapping between any pair of measurement scales with known or estimable individual-level relationships. We verify and contrast our methods using simulation studies and trials from our data set which report outcomes on multiple scales. We find that all methods recreate mean values with reasonable accuracy, but for standard deviations, optimization outperforms the other methods. However, the optimization method is more likely to underestimate standard deviations and is vulnerable to non-convergence.

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two different approaches: (1) manually by human reviewers, and (2) automatically by the RobotReviewer. The manual assessment was based on two groups independently, with two additional rounds of verification. The agreement between RobotReviewer and humans was measured via the concordance rate and Cohen's kappa statistics, based on the comparison of binary classification of the risk of bias (low vs. high/unclear) as restricted by RobotReviewer. The concordance rates varied by domain, ranging from 63.07% to 83.32%. Cohen's kappa statistics showed a poor agreement between humans and RobotReviewer for allocation concealment (κ = 0.25, 95% CI: 0.21–0.30), blinding of outcome assessors (κ = 0.27, 95% CI: 0.23–0.31); While moderate for random sequence generation (κ = 0.46, 95% CI: 0.41–0.50) and blinding of participants and personnel (κ = 0.59, 95% CI: 0.55–0.64). The findings demonstrate that there were domain-specific differences in the level of agreement between RobotReviewer and humans. We suggest that it might be a useful auxiliary tool, but the specific manner of its integration as a complementary tool requires further discussion.