Research Synthesis Methods最新文献

Understanding causality is crucial for social scientific research to develop strong theories and inform practice. However, explicit discussion of causality is often lacking in social science literature due to ambiguous causal language. This paper introduces a text mining model fine-tuned to extract causal sentences from full-text social science papers. A dataset of 529 causal and 529 non-causal sentences manually annotated from the Cooperation Databank (CoDa) was curated to train and evaluate the model. Several pre-trained language models (BERT, SciBERT, RoBERTa, LLAMA, and Mistral) were fine-tuned on this dataset and general-purpose causality datasets. Model performance was evaluated on held-out social science and general-purpose test sets. Results showed that fine-tuning transformer models on the social science dataset significantly improved causal sentence extraction, even with limited data, compared to the models fine-tuned only on the general-purpose data. Results indicate the importance of domain-specific fine-tuning and data for accurately capturing causal language in academic writing. This automated causal sentence extraction method enables comprehensive, large-scale analysis of causal claims across the social sciences. By systematically cataloging existing causal statements, this work lays the foundation for further research to uncover the mechanisms underlying social phenomena, inform theory development, and strengthen the methodological rigor of the field.

The current meta-analytic structural equation modeling (MASEM) techniques cannot properly deal with cases where there are multiple effect sizes available for the same relationship from the same study. Existing applications either treat these effect sizes as independent, randomly select one effect size amongst many, or create an average effect size. None of these approaches deal with the inherent dependency in effect sizes, and either leads to biased estimates or loss of information and power. An alternative technique is to use univariate three-level modeling in the two-stage approach to model these dependencies. These different strategies for dealing with dependent effect sizes in the context of MASEM have not been previously compared in a simulation study. This study aims to compare the performance of these strategies across different conditions; varying the number of studies, the number of dependent effect sizes within studies, the correlation between the dependent effect sizes, the magnitude of the path coefficient, and the between-studies variance. We examine the relative bias in parameter estimates and standard errors, coverage proportions of confidence intervals, as well as mean standard error and power as measures of efficiency. The results suggest that there is not one method that performs well across all these criteria, pointing to the need for better methods.

Aim(s): To describe a sequential mixed methods review method that prioritized synthesized qualitative evidence from primary studies to explain the complexities of older persons with multiple chronic conditions' unplanned readmission experiences.

Background: Segregated mixed methods review studies frequently prioritize quantitative evidence synthesis to examine the effectiveness of interventions; utilizing qualitative evidence to explain quantitative data. There is a lack of guidance about how to prioritize qualitative evidence.

Results: Five procedural steps were developed to prioritize qualitative evidence synthesis. In Step 1, research questions were developed. In Step 2, databases were searched, studies were mapped to their method (qualitative or quantitative) and appraised. In Step 3, meta-synthesis and applied thematic analysis were used to synthesize extracted qualitative evidence about the psychosocial processes and factors that influenced unplanned readmission. In Step 4, quantitative evidence was synthesized using vote counting to determine the factors influencing unplanned readmission. In Step 5, a matrix was used to compare, determine the agreement between the qualitative and quantitative evidence, juxtapose findings, and uphold validity. Factors were mapped to the model of psychosocial processes and analytic themes.

Conclusion: Prioritizing qualitative evidence synthesis in a mixed methods review study prioritizes participants' experiences, perspectives, and voices to understand complex clinical problems from participants who experienced the event. Synthesizing and integrating evidence facilitates the construction of holistic new understandings about phenomenon and expands mixed methods systematic review methods.

Implications: Prioritizing patients' perspectives is useful for developing new client-centered interventions, establishing best practices for future reviews, generating theories, and expanding research methods.

Conventional meta-analyses (both fixed and random effects) of correlations are biased due to the mechanical relationship between the estimated correlation and its standard error. Simulations that are closely calibrated to match actual research conditions widely seen across correlational studies in psychology corroborate these biases and suggest two solutions: UWLS₊₃ and HS. UWLS₊₃ is a simple inverse-variance weighted average (the unrestricted weighted least squares) that adjusts the degrees of freedom and thereby reduces small-sample bias to scientific negligibility. UWLS₊₃ as well as the Hunter and Schmidt approach (HS) are less biased than conventional random-effects estimates of correlations and Fisher's z, whether or not there is publication selection bias. However, publication selection bias remains a ubiquitous source of bias and false-positive findings. Despite the relationship between the estimated correlation and its standard error in the absence of selective reporting, the precision-effect test/precision-effect estimate with standard error (PET-PEESE) nearly eradicates publication selection bias. Surprisingly, PET-PEESE keeps the rate of false positives (i.e., type I errors) within their nominal levels under the typical conditions widely seen across psychological research whether there is publication selection bias, or not.

While the Institute of Education Science's ERIC is often recommended for comprehensive literature searching in the field of education, there are several other specialized education databases to discover education literature. This study investigates journal coverage overlaps between four specialized education databases: Education Source (EBSCO), Education Database (ProQuest), ERIC (Institute of Education Sciences), and Educator's Reference Complete (Gale). Out of a total of 4,695 unique journals analyzed, there are 2,831 journals uniquely covered by only one database, as well as many journals covered by only two or three databases. Findings show that evidence synthesis projects and literature reviews benefit from the careful selection of multiple specialized education databases and that ERIC is insufficient as the primary education database for comprehensive searching in the field.

The number of published systematic reviews has increased over the last years, with a non-negligible proportion displaying methodological concerns. We aimed to develop and evaluate a tool to assess the reported methodological quality of medical systematic reviews. The developed tool (ReMarQ) consists of 26 dichotomous items. We applied an item response theory model to assess the difficulty and discrimination of the items and decision tree models to identify those items more capable of identifying systematic reviews with higher reported methodological quality. ReMarQ was applied to a representative sample of medical systematic reviews (excluding those published in the Cochrane Database of Systematic Reviews) to describe their methodological quality and identify associated factors. We assessed 400 systematic reviews published between 2010 and 2020, of which 196 (49.0%) included meta-analysis. The most discriminative items were (i) conducting a risk of bias assessment, (ii) having a published protocol and (iii) reporting methods for solving disagreements. More recent systematic reviews (adjusted yearly RR=1.03; 95%CI=1.02 -1.04, p<0.001) and those with meta-analysis (adjusted RR=1.34; 95%CI=1.25 -1.43, p<0.001) were associated with higher reported methodological quality. Such an association was not observed with the journal impact factor. The items most frequently fulfilled were (i) reporting search dates, (ii) reporting bibliographic sources and (iii) searching multiple electronic bibliographic databases. ReMarQ, consisting of dichotomous items and whose application does not require subject content expertise, may be important (i) in supporting an efficient quality assessment of systematic reviews and (ii) as the basis of automated processes to support that assessment.

Data visualization is crucial for effectively communicating knowledge in meta-analysis. However, existing visualization methods in meta-analysis have predominantly focused on quantitative aspects, such as forest plots and funnel plots, thereby neglecting qualitative information that is equally important for end-users in science, policy, and practice. We introduce a framework consisting of a series of visualization toolkits designed to enrich meta-analyses by borrowing approaches from other research synthesis methods, including systematic evidence mapping (scoping reviews), bibliometrics (bibliometric analysis), and alternative impact metric analysis. These "enrichment" toolkits aim to facilitate the synthesis of both quantitative and qualitative evidence, along with the assessment of the academic and nonacademic influences of the meta-analytic evidence base. While the meta-analysis yields quantitative insights, the enrichment analyses, and visualizations provide user-friendly summaries of qualitative information on the evidence base. For example, a systematic evidence map can visualize study characteristics, unraveling knowledge gaps and methodological differences. Bibliometric analysis offers a visual assessment of the nonindependent evidence, such as hyper-dominant authors and countries, and funding sources, potentially informing the risk of bias. Alternative impact metric analysis employs alternative metrics to gauge societal influence and research translation (e.g., policy and patent citations) of studies in the meta-analysis. We provide a dedicated webpage showcasing sample visualizations and providing step-by-step implementation in open-source software R (https://yefeng0920.github.io/MA_Map_Bib/). Additionally, we offer a guide on leveraging three commercially free large language models (LLMs) to help adapt the sample script, enabling users with less R coding experience to visualize their own meta-analytic evidence base.

The product Bayes factor (PBF) synthesizes evidence for an informative hypothesis across heterogeneous replication studies. It can be used when fixed- or random effects meta-analysis fall short. For example, when effect sizes are incomparable and cannot be pooled, or when studies diverge significantly in the populations, study designs, and measures used. PBF shines as a solution for small sample meta-analyses, where the number of between-study differences is often large relative to the number of studies, precluding the use of meta-regression to account for these differences. Users should be mindful of the fact that the PBF answers a qualitatively different research question than other evidence synthesis methods. For example, whereas fixed-effect meta-analysis estimates the size of a population effect, the PBF quantifies to what extent an informative hypothesis is supported in all included studies. This tutorial paper showcases the user-friendly PBF functionality within the bain R-package. This new implementation of an existing method was validated using a simulation study, available in an Online Supplement. Results showed that PBF had a high overall accuracy, due to greater sensitivity and lower specificity, compared to random-effects meta-analysis, individual participant data meta-analysis, and vote counting. Tutorials demonstrate applications of the method on meta-analytic and individual participant data. The example datasets, based on published research, are included in bain so readers can reproduce the examples and apply the code to their own data. The PBF is a promising method for synthesizing evidence for informative hypotheses across conceptual replications that are not suitable for conventional meta-analysis.

Psychological interventions are complex in nature and have been shown to benefit various clinical outcomes. Gaining insight into current practices would help identify specific aspects that need improvement to enhance the quality of network meta-analysis (NMA) in this field. This scoping review aimed to explore methodological approaches in the NMA of psychological interventions. We searched PubMed, EMBASE, and Cochrane CENTRAL in September 2023. We included NMAs of psychological interventions of randomized controlled trials that reported clinical outcomes. Three independent researchers assessed the eligibility and extracted relevant data. The findings were presented using descriptive statistics. Of the 1827 articles identified, 187 studies were included. Prior protocol registration was reported in 130 studies (69.5%). Forty-six studies (24.6%) attempted to search for gray literature. Ninety-four studies (50.3%) explicitly assessed transitivity. Nearly three-quarters (143 studies, 76.5%) classified treatment nodes by the type of psychological intervention, while 13 studies (7.0%) did so by lumping different intervention types into more broader intervention classes. Seven studies (3.7%) examined active components of the intervention using component NMA. Only three studies (1.6%) classified interventions based on factors affecting intervention practices, specifically intensity, provider, and delivery platform. Meanwhile, 29 studies (15.5%) explored the influential effects of these factors using meta-regression, subgroup analysis, or sensitivity analysis. The certainty of evidence was assessed in 80 studies (42.8%). The methodological approach in NMAs of psychological interventions should be improved, specifically in classifying psychological interventions into treatment nodes, exploring the effects of intervention-related factors, and assessing the certainty of evidence.