Section Editor’s Note: Insights into the Generalizability of Findings from Experimental Evaluations

IF 1.1 3区 社会学 Q2 SOCIAL SCIENCES, INTERDISCIPLINARY American Journal of Evaluation Pub Date : 2022-03-01 DOI:10.1177/10982140221075092
Laura R. Peck
{"title":"Section Editor’s Note: Insights into the Generalizability of Findings from Experimental Evaluations","authors":"Laura R. Peck","doi":"10.1177/10982140221075092","DOIUrl":null,"url":null,"abstract":"As noted in my Editor’s Note to the Experimental Methodology Section of the American Journal of Evaluation’s (2020) Volume 40, Issue 4, experimental evaluations—where research units, such as people, schools, classrooms, and neighborhoods are randomly assigned to a program or to a control group—are often criticized for having limited external validity. In evaluation parlance, external validity refers to the ability to generalize results to other people, places, contexts, or times beyond those on which the evaluation focused. Evaluations—whether using an experimental design or not—are commonly conducted in a single site or a selected set of sites, either because that site is of particular interest or for convenience. Those special circumstances can mean that those sites—or the people within them—are not representative of a broader population of interest. In turn, the evaluation results may be useful only for assessing those people and places and not for predicting how a similar intervention might generate similar results for other people in other places. The good news, however, is that research and design innovations over the past several years have focused on how to overcome this criticism, making experimental evaluations’ results more useful for informing policy and program decisions (e.g., Bell & Stuart, 2016; Tipton & Olsen, 2018). Efforts for improving the external validity of experiments fall into two camps: design and analysis. Improving external validity through design means explicitly engaging a sample that is representative of a clearly identified target population. Although doing so is not common, particularly at the national level, some experiments have been successful at engaging a representative set of sites. The U.S. Department of Labor’s National Job Corps Study (e.g., Schochet, Burghardt & McConnell, 2006), the U.S. Department of Health and Human Services’ Head Start Impact Study (Puma et al., 2010), and the U.S. Social Security Administration’s Benefit Offset National Evaluation (Gubits et al., 2018) are three major evaluations that successfully recruited a nationally representative sample so that the evaluation results would be nationally generalizable. A simple, random selection of sites is the most straightforward way to ensure this representativeness and the generalizability of an evaluation’s results. In practice, however, that can be anything but simple. Even if an evaluation team randomly samples a site to participate, that site still needs to agree to participate; and if it does not, then the sample is no longer random.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.1000,"publicationDate":"2022-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Journal of Evaluation","FirstCategoryId":"90","ListUrlMain":"https://doi.org/10.1177/10982140221075092","RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"SOCIAL SCIENCES, INTERDISCIPLINARY","Score":null,"Total":0}
引用次数: 0

Abstract

As noted in my Editor’s Note to the Experimental Methodology Section of the American Journal of Evaluation’s (2020) Volume 40, Issue 4, experimental evaluations—where research units, such as people, schools, classrooms, and neighborhoods are randomly assigned to a program or to a control group—are often criticized for having limited external validity. In evaluation parlance, external validity refers to the ability to generalize results to other people, places, contexts, or times beyond those on which the evaluation focused. Evaluations—whether using an experimental design or not—are commonly conducted in a single site or a selected set of sites, either because that site is of particular interest or for convenience. Those special circumstances can mean that those sites—or the people within them—are not representative of a broader population of interest. In turn, the evaluation results may be useful only for assessing those people and places and not for predicting how a similar intervention might generate similar results for other people in other places. The good news, however, is that research and design innovations over the past several years have focused on how to overcome this criticism, making experimental evaluations’ results more useful for informing policy and program decisions (e.g., Bell & Stuart, 2016; Tipton & Olsen, 2018). Efforts for improving the external validity of experiments fall into two camps: design and analysis. Improving external validity through design means explicitly engaging a sample that is representative of a clearly identified target population. Although doing so is not common, particularly at the national level, some experiments have been successful at engaging a representative set of sites. The U.S. Department of Labor’s National Job Corps Study (e.g., Schochet, Burghardt & McConnell, 2006), the U.S. Department of Health and Human Services’ Head Start Impact Study (Puma et al., 2010), and the U.S. Social Security Administration’s Benefit Offset National Evaluation (Gubits et al., 2018) are three major evaluations that successfully recruited a nationally representative sample so that the evaluation results would be nationally generalizable. A simple, random selection of sites is the most straightforward way to ensure this representativeness and the generalizability of an evaluation’s results. In practice, however, that can be anything but simple. Even if an evaluation team randomly samples a site to participate, that site still needs to agree to participate; and if it does not, then the sample is no longer random.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
编者按:对实验评估结果的可推广性的见解
正如我在《美国评估杂志》(2020)第40卷第4期实验方法论部分的编者注中所指出的那样,实验评估——研究单位,如人、学校、教室和社区,被随机分配到一个项目或对照组——经常被批评为外部有效性有限。在评估术语中,外部效度指的是将结果推广到评估所关注的其他人、地点、背景或时间之外的能力。评估——无论是否使用实验设计——通常在单个地点或选定的一组地点进行,要么是因为该地点特别有趣,要么是为了方便。这些特殊情况可能意味着这些地点——或其中的人们——并不能代表更广泛的兴趣人群。反过来,评估结果可能只对评估这些人和地方有用,而不是预测类似的干预如何对其他地方的其他人产生类似的结果。然而,好消息是,过去几年的研究和设计创新一直专注于如何克服这种批评,使实验评估结果更有助于为政策和项目决策提供信息(例如,Bell & Stuart, 2016;Tipton & Olsen, 2018)。提高实验外部有效性的努力分为两大阵营:设计和分析。通过设计提高外部效度意味着明确地参与一个样本,代表一个明确确定的目标人群。虽然这样做并不普遍,特别是在国家一级,但一些试验在吸引一批有代表性的场址方面取得了成功。美国劳工部的全国就业队伍研究(例如,Schochet, Burghardt & McConnell, 2006年),美国卫生与公众服务部的领先影响研究(Puma等人,2010年)和美国社会保障局的福利抵消国家评估(Gubits等人,2018年)是三个主要评估,它们成功地招募了具有全国代表性的样本,从而使评估结果具有全国普遍性。一个简单的,随机选择的地点是最直接的方式,以确保这种代表性和概括性的评估结果。然而,在实践中,这一点都不简单。即使一个评估小组随机抽取一个站点参与,该站点仍然需要同意参与;如果不是,那么样本就不再是随机的。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
American Journal of Evaluation
American Journal of Evaluation SOCIAL SCIENCES, INTERDISCIPLINARY-
CiteScore
4.40
自引率
11.80%
发文量
39
期刊介绍: The American Journal of Evaluation (AJE) publishes original papers about the methods, theory, practice, and findings of evaluation. The general goal of AJE is to present the best work in and about evaluation, in order to improve the knowledge base and practice of its readers. Because the field of evaluation is diverse, with different intellectual traditions, approaches to practice, and domains of application, the papers published in AJE will reflect this diversity. Nevertheless, preference is given to papers that are likely to be of interest to a wide range of evaluators and that are written to be accessible to most readers.
期刊最新文献
The Garden of Evaluation Approaches From the Section Editors: Teaching & Learning Section Vision: Innovate, Evaluate, Disseminate From the Co-Editors: Evolving Evaluation Theory, Methods, and Practice Application of Multi-Attribute Utility Analysis as a Methodological Framework in Academic–Clinical Partnership Evaluation A Protocol for Participatory Data Use
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1