Evaluation Review最新文献

Background: While randomized controlled trials (RCTs) are typically considered the gold standard of program evaluation, they are infrequently chosen by public sector leaders, defined as government and nonprofit decision-makers, when an impact evaluation is required. Objectives: This study provides descriptive evidence on RCT aversion among public sector leaders and attempts to understand what factors affect their likelihood of choosing RCTs for impact evaluations. Research Design: The authors ask if public sector leaders follow similar preference patterns found among non-public sector leaders when choosing either an RCT or a quasi-experimental design and use a survey experiment to determine which factors affect the RCT choice. Subjects: The study sample includes 2050 public sector leaders and a comparison group of 2060 respondents who do not lead public sector organizations. Measures: The primary outcome measure is selecting an RCT as the preferred evaluation option. Results: When asked to make a decision about an impact evaluation, the majority of people do not choose an RCT. While also averse to RCTs, public sector leaders are about 13% more likely to prefer a RCT to a quasi-experimental evaluation compared to the general population. Public sector leaders are less likely to use RCTs for evaluations of more intense interventions, potentially because they are perceived to be superior to the options available for the control group. Conclusion: Funders should be aware that when given a choice, public sector leaders prefer other options to RCTs. Greater awareness of the benefits of RCTs could increase their use in the public sector.

As found in behavioral decision theory, venture capitalists (VCs) rely on heuristics and bias, owing to their bounded rationality, either by limited alternatives or information and resources. India's booming startup scene challenges VCs in decision-making owing to information overload from numerous evolving ventures, which hinders informed judgment. VC investment behavior, due diligence, and cognitive factors related to decision-making have always drawn the attention of researchers. We provide an alternative approach for an optimal decision by VCs by identifying the attributes that influence investment or funding decisions at an early stage of a venture in tech-based industries. Through a literature review, we identify eight attributes, both on internal and external criteria, that venture investors consider when making investment decisions. Based on interviews with 20 experts, we further identify eight key tech-based sectors. Using grey system theory, we then determine the rankings of eight tech startups for investors' early-stage investment decisions. This study presents a linguistic variable-based approach of grey numbers to decide weights and ratings, the grey possibility degree to compare and rank different tech startups, and based on the results, suggests the ideal tech startup. We find that agritech ranks first; thus, investors should prefer venturing into such startups for early-stage investment. E-commerce and edutech ranked second and third, respectively, followed by electric vehicle infrastructure, insurtech, fintech, space tech, and software as a service.

Maternal, neonatal, and child health play crucial roles in achieving the objectives of Sustainable Development Goal (SDG) 2030, particularly in promoting health and wellbeing. However, maternal, neonatal, and child services in Moroccan public hospitals face challenges, particularly concerning mortality rates and inefficient resource allocation, which hinder optimal outcomes. This study aimed to evaluate the operational effectiveness of 76 neonatal and child health services networks (MNCSN) within Moroccan public hospitals. Using Data Envelopment Analysis (DEA), we assessed technical efficiency (TE) employing both Variable Returns to Scale for inputs (VRS-I) and outputs (VRS-O) orientation. Additionally, the Tobit method (TM) was utilized to explore factors influencing inefficiency, with hospital, doctor, and paramedical staff considered as inputs, and admissions, cesarean interventions, functional capacity, and hospitalization days as outputs. Our findings revealed that VRS-I exhibited a higher average TE score of 0.76 compared to VRS-O (0.23). Notably, the Casablanca-Anfa MNCSN received the highest referrals (30) under VRS-I, followed by the Khemisset MNCSN (24). In contrast, under VRS-O, Ben Msick, Rabat, and Mediouna MNCSN each had three peers, with 71, 22, and 17 references, respectively. Moreover, the average Malmquist Index under VRS-I indicated a 7.7% increase in productivity over the 9-year study period, while under VRS-O, the average Malmquist Index decreased by 8.7%. Furthermore, doctors and functional bed capacity received the highest Tobit model score of 0.01, followed by hospitalization days and cesarean sections. This study underscores the imperative for policymakers to strategically prioritize input factors to enhance efficiency and ensure optimal maternal, neonatal, and child healthcare outcomes.

Cooperation between employees in a company is an important input to firm performance. This study examines how a manager's cooperative behavior and the visibility of this behavior affect the cooperation amongst employees, and subsequently firm performance. To do so, we conducted a field experiment with managers and their employees from 320 Vietnamese small and micro firms to determine the impact of a manager's leading by example (LBE) on employees' behavior, corporate culture, and firm performance. Both managers and employees participated in a Public Good experiment which aimed to elicit an individual cooperative behavior. Noteworthy is that the decision made by a manager in the experiment was given as an example to employees before they made decision in that same experiment. We considered that the example of cooperation by managers in the Public Good experiment communicated a powerful signal to the employees regarding the importance of fostering cooperation in the workplace. Such a signal by the manager, who is at the top in the organizational hierarchy, would impact their employees' behavior in the workplace and firm's outcomes beyond the experiment. Interestingly, we found that concealing a manager's identity from their employees enhances the impacts of LBE.

Recent attention to the causal identification of spending impacts provides improved estimates of spending outcomes in a variety of circumstances, but the estimates are substantially different across studies. Half of the variation in estimated funding impact on test scores and over three-quarters of the variation of impacts on school attainment reflect differences in the true parameters across study contexts. Unfortunately, inability to describe the circumstances underlying effective school spending impedes any attempts to generalize from the extant results to new policy situations. The evidence indicates that how funds are used is crucial to the outcomes, but such factors as targeting of funds or court interventions fail to explain the existing pattern of results.

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

This chapter begins with an overview of recent developments that have encouraged and facilitated greater use of research syntheses, including Meta-Analysis, to guide public policy and practice in education, workforce development, and social services. It discusses the role of Meta-Analysis for improving knowledge of the effectiveness of programs, policies, and practices and the applicability and generalizability of that knowledge to conditions other than those represented by the study samples and settings. The chapter concludes with recommendations for improving the potential of Meta-Analysis to accelerate knowledge development through changing how we design, conduct, and report findings of individual studies to maximize their usefulness in Meta-Analysis as well as how we produce and report Meta-Analysis findings. The paper includes references to resources supporting the recommendations.

Assessing the transferability of lessons from social research or evaluation continues to raise challenges. Efforts to identify transferable lessons can be based on two different forms of argumentation. The first draws upon statistics and causal inferences. The second involves constructing a reasoned case based on weighing up different data collected along the causal chain from designing to delivery. Both approaches benefit from designing research based upon existing evidence and ensuring that the descriptions of the programme, context, and intended beneficiaries are sufficiently rich. Identifying transferable lessons should not be thought of as a one-off event but involves contributing to the iterative and learning of a scientific community. To understand the circumstances under which findings can be confidently transferred, we need to understand: (1) How far and why outcomes of interest have multiple, interacting and fluctuating causes. (2) The program design and implementation capacity. (3) Prior knowledge and causal landscapes (and how far these are included in the theory of change). (4) New and relevant knowledge; what can we learn in our 'disputatious community of truth seekers'.

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.