Evaluation Review最新文献

Strategies for ensuring the generalizability of evaluation results often focus on building methodologically sound research designs. However, these strategies may fall short when the challenge lies as much in uncovering causal patterns than in applying them to uncertain contexts. This article proposes a set of pragmatic strategies to address generalizability issues in complex evaluations. These strategies address important questions such as: What needs to be generalized, for whom and for what purpose? What assumptions underlying the intervention could be generalized? How can approaches and methods be selected to support a generalization objective? How can potential users be involved to support the process of generalization? We illustrate these strategies using the example of a series of evaluations of bottom-up socio-ecological transition initiatives in France between 2013 and 2020. We suggest that these strategies can enhance policy learning and better equip policy stakeholders to navigate complex situations.

This introduction to the second special issue of Evaluation review on external validity and generalizability opens a dialogue between different ways to think about generalizability in program evaluation. It argues that generalizability in impact evaluation fundamentally is about inferring some form of causality at a level broader than the specific circumstances of the initial study or studies from which these inferences are drawn. The question, then, is about how one apprehends causality: in other words, what is being generalized and how? The first special issue mainly relied on a counterfactual conception of causality, embodied by experimental and quasi-experimental methods, that aimed at impact measurement. The articles in this volume, drawing on mixed methods, also mobilize generative and configurational causal inferences to provide further levers of generalizability, focusing on how the impact is produced. The introduction insists on the specific input of qualitative methods in this respect, as theorized by grounded theory.

This paper presents a methodological advancement by integrating Qualitative Comparative Analysis (QCA) and Process Tracing (PT) in the evaluation of training transfer effectiveness in Flemish SMEs. This multimethod approach leverages the strengths of both QCA and PT to enhance internal and external validity, offering a robust framework for capturing the conditions and causal mechanisms underlying policy interventions. By sequentially applying QCA to identify necessary and sufficient conditions and PT to unpack the causal processes, the study provides a comprehensive analysis that addresses both "what works" and "how it works." Our findings demonstrate that combining these methods allows for more nuanced insights into the effectiveness of training programs, ultimately contributing to the empirical validation of policy theories and the development of evidence-based interventions. This research underscores the potential of multimethod evaluations to produce more reliable and generalizable results, thereby offering valuable guidance for evaluators and policymakers seeking to enhance the impact of their programs.

This article reflects on how the Independent Evaluation Group (IEG) of the World Bank Group grapples with the issue of generalizability in conducting large-scale and complex development evaluations. It discusses a practical framework IEG uses to meet three needs: methodological, institutional, and didactic. IEG evaluations are aimed at informing significant organizational or strategic decisions encompassing broad scopes, assessing extensive portfolios of interventions by the organization across diverse contexts and spanning multiple years. These evaluations are inherently multimethod and need to bridge various logics of generalization. They seek to influence decision-makers, such as boards of directors or senior managerial teams, with the objective of guiding pivotal moments in the organization's trajectory; ensuring accountability for learning, results, or budget expenditures; and synthesizing substantial evidence to distill key success or failure factors for future strategic planning. The defensibility of the methodological scaffolding is paramount to the credibility of the evaluations. The article discusses the challenges inherent in such evaluations, including the need to generate findings that are valid at multiple levels of analysis and the reliance on multitiered mixed-methods approaches. It examines the use of a practical framework to bridge methodological principles and real-world challenges involved in evaluation to inform the design and implementation of large-scale evaluations. The framework is illustrated with examples from IEG's evaluations, and the article explores how practitioners and researchers can apply the framework in other settings to enhance the generalizability of their findings.

Mixed methods research, that is, research that integrates qualitative and quantitative methods, has become increasingly popular in program evaluation because of its potential for understanding complex interventions. Despite recent constructive and fruitful developments that have led to the consolidation of mixed methods as a distinctive methodology, fundamental methodological issues such as generalization have received little attention. The purpose of this paper is to provide a critical reflection on how the concept of generalization has been used in mixed methods research. The paper is structured into four main parts. First, we discuss the relevance of external validity and mixed methods research in impact evaluation. Second, we summarize how generalization is conceptualized in mixed methods research. Third, we present the results of a literature review on generalization practices in mixed methods research. Finally, we conclude with a discussion of threats to and strategies for enhancing generalization in mixed methods research.

This paper evaluates the impact of racial and low-income quotas on the academic performance of senior students in Brazilian colleges and universities. Using longitudinal data from Brazil's Higher Education Census and the National Examination of Student Performance (ENADE), and employing a fixed effects approach, the study examines the influence of these quotas on student outcomes. The results show that neither racial nor low-income quotas significantly impact the academic performance of either quota or non-quota students. This finding holds across different groups of majors, indicating that the inclusion of quota students does not detract from overall student achievement.

Negative menstruation experiences adversely affect schoolgirls' social participation, education, and overall health, yet comprehensive menstrual health interventions are limited in Tanzanian schools. This study evaluated the feasibility, acceptability, cost, and potential impact mechanisms of a comprehensive school-based menstrual, sexual, and reproductive health (MSRH) intervention. The intervention, piloted in four secondary schools, included education sessions for girls and boys, pain management, distribution of menstrual kits (reusable pads and menstrual cups), WASH improvements, and stakeholder engagement. A mixed-methods process evaluation assessed acceptability, fidelity, cost, context, and potential impact mechanisms using qualitative methods (in-depth interviews, focus group discussions, and WASH observations) and quantitative methods (survey questionnaires, structured observation of education sessions, costing, and monitoring data). The intervention was well received by students, teachers, and local government authorities; MSRH education reached 86% of schoolgirls and 72% of schoolboys, while over 93% of girls received menstrual kits. Total implementation cost across the four schools was 111,347,467 TZS (38,003 GBP), approximately 39.05 GBP per student, lower than comparable initiatives in East Africa. Findings indicate the intervention is feasible and acceptable in school settings and can inform future menstrual health and hygiene programs, though further research is needed to assess broader effectiveness and sustainability.

There has been an increasing call for social justice, not just in American society but also within the evaluation profession. Scholars and practitioners like Donna Mertens, Stafford Hood, Karen Kirkhart, Veronica Thomas, and many others have led the movement within the evaluation profession to view evaluation practice as a vehicle for social justice. However, descriptions of social justice-oriented evaluation in the literature tend to lack sufficient detail, leaving many evaluators to wonder exactly what social justice-oriented evaluation is. This article presents a summary of the literature on social justice-oriented evaluation, providing a much-needed description of the approach. This paper also introduces a taxonomy of social justice-oriented evaluation (T-SJOE) practices, developed from the results of the literature review, and presents a framework of social justice-oriented evaluation practices as they are currently discussed in the evaluation literature. T-SJOE is not prescriptive or presumed complete but is presented as a tool for self-reflection and as a resource when planning evaluations. Further development of the T-SJOE is expected and encouraged.

Multilevel mediation analyses are vital for testing and refining theories of action in program evaluations. Prior literature and established guidelines have repeatedly underscored the need to prospectively incorporate mediation considerations when designing a study even when mediation is considered a secondary or exploratory aim. Despite the critical role mediation analyses play across disciplines, the available empirical resources to facilitate effective and efficient design of such studies are limited. This study contributes to this gap by developing empirical estimates of design parameter values for literacy programs implemented within classrooms by teachers. Using data from two large studies with thousands of students and hundreds of teachers and schools, we develop empirical guidelines for studies using a three-level structure (students, teachers, and schools) for a diverse set of school contexts (e.g., urban and geographic regions) and mediators (e.g., teacher knowledge, instruction, affect, attitudes, and classroom climate). The results suggested considerable variation in the clustering of teacher mediators and student outcomes within schools and that such clustering is dependent on the variable type (e.g., instruction, knowledge, climate, and literacy achievement) and study context. The findings underscore the importance of context-, mediator-, and outcome-specific design parameter values in planning studies.

Although the United States has safe haven laws, which allow mothers to safely relinquish their babies to designated safety points, rates of infant mortality remain significantly higher in the United States than other similarly developed countries. The current study is seeking to explore the state-level association between safe haven laws and infant mortality in the United States utilizing a legal epidemiological approach. Several sources of publicly available data were combined to examine the state-level association between safe haven laws and rates of infant mortality. A backward stepwise regression was used to determine whether certain safe haven laws significantly predicted rates of infant mortality, while controlling for demographic variables. After controlling for demographic variables including rates of health insurance and poverty, safe haven laws stating that only the mother can relinquish a child, laws that protect parents from criminal liability, and laws requiring the provider to provide legal information and referrals were associated with infant mortality, with the total model accounting for 70.1% of variance in infant mortality. These results have important implications for policymakers considering the reform of the safe haven laws. It is especially important to evaluate the effectiveness and origins of safe haven laws in the wake of the overturn of Roe v. Wade. Future studies should longitudinally examine how changes in safe haven laws lead to changes in rates of infant mortality.