New Directions for Evaluation最新文献

This paper describes the theoretical and conceptual frameworks used to guide the site-level evaluations of Building Infrastructure Leading to Diversity (BUILD) programs, part of the Diversity Program Consortium (DPC), funded by the National Institutes of Health. We aim to provide an understanding of which theories informed the evaluation work of the DPC and how the frameworks guiding BUILD site-level evaluations are conceptually aligned with one another and with the consortium-level evaluation.

BUilding Infrastructure Leading to Diversity (BUILD), an initiative of the National Institutes of Health (NIH), provides grants to undergraduate institutions to implement and study innovative approaches to engaging and retaining students from diverse backgrounds in biomedical research. The NIH awarded BUILD grants to 10 higher education institutions in multiple states, including funding for local evaluations. This chapter presents findings from an online survey and interviews with 15 local evaluators from nine of the 10 BUILD sites. Participants shared their perspectives on the role of professional local evaluators in national evaluations, ideal national-local multisite evaluation partnerships, and the ways that funders can support these partnerships to maximize impact. They argued for customized technical assistance and other support for local evaluations; the importance of including local results in national evaluation findings; the value of local evaluators' subject-matter expertise; and the potential for funders to act as central organizing entities in national-local evaluation partnerships.

Advancing diversity in the biomedical research workforce is critical to the ability of the National Institutes of Health (NIH) to achieve its mission. The NIH Diversity Program Consortium is a unique, 10-year program that builds upon longstanding training and research capacity-building activities to promote workforce diversity. It was designed to rigorously evaluate approaches to enhancing diversity in the biomedical research workforce at the student, faculty, and institutional level. In this chapter we describe (a) the program's origins, (b) the consortium-wide evaluation, including plans, measures, challenges, and solutions, and (c) how lessons learned from this program are being leveraged to strengthen NIH research-training and capacity-building activities and evaluation efforts.

While guidance on how to design rigorous evaluation studies abounds, prescriptive guidance on how to include critical process and context measures through the construction of exposure variables is lacking. Capturing nuanced intervention dosage information within a large-scale evaluation is particularly complex. The Building Infrastructure Leading to Diversity (BUILD) initiative is part of the Diversity Program Consortium, which is funded by the National Institutes of Health. It is designed to increase participation in biomedical research careers among individuals from underrepresented groups. This chapter articulates methods employed in defining BUILD student and faculty interventions, tracking nuanced participation in multiple programs and activities, and computing the intensity of exposure. Defining standardized exposure variables (beyond simple treatment group membership) is crucial for equity-focused impact evaluation. Both the process and resulting nuanced dosage variables can inform the design and implementation of large-scale, diversity training program, outcome-focused, evaluation studies.

The National Institutes of Health (NIH) created the Building Infrastructure Leading to Diversity (BUILD) initiative to incentivize undergraduate institutions to create innovative approaches to increasing diversity in biomedical research, with the ultimate goal of diversifying the NIH-funded research enterprise. Initiatives such as BUILD involve designing and implementing programs at multiple sites that share common objectives. Evaluation of initiatives like this often includes statistical analyses that combine data across sites to estimate the program's impact on particular outcomes. Meta-analysis is a statistical technique for combining effect estimates from different studies to obtain a single overall effect estimate and to estimate heterogeneity across studies. However, it has not been commonly applied to evaluate the impact of a program across multiple different sites. In this chapter, we use the BUILD Scholar program-one component of the broader initiative-to demonstrate the application of meta-analysis to combine effect estimates from different sites of a multisite initiative. We analyze three student outcomes using a typical "single-stage" modeling approach and a meta-analysis approach. We show how a meta-analysis approach can provide more nuanced information about program impacts on student outcomes and thus can help support a robust evaluation.