Pub Date : 2018-11-10DOI: 10.23943/princeton/9780691183138.003.0013
Dean S. Karlan, J. Appel
This concluding chapter offers some guide on how to run a field study. First, researchers should think about where, when, and with whom they will run their experiment, and make sure these parameters fit the underlying idea or theory they intend to test. Second, every question in a survey should have a purpose. Researchers should be mindful that subtle features of a survey like response scales and order of questions can influence the results. Third, researchers should make sure that their implementing partner understands what it will take to conduct a research. Fourth, researchers should make an intentional decision about how, and how much, to incorporate technology into their survey. Fifth, researchers should not assume people will sign up to receive a program or service. They should find out directly whenever possible by piloting or otherwise gauging demand for their intervention.
{"title":"Conclusion","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/princeton/9780691183138.003.0013","DOIUrl":"https://doi.org/10.23943/princeton/9780691183138.003.0013","url":null,"abstract":"This concluding chapter offers some guide on how to run a field study. First, researchers should think about where, when, and with whom they will run their experiment, and make sure these parameters fit the underlying idea or theory they intend to test. Second, every question in a survey should have a purpose. Researchers should be mindful that subtle features of a survey like response scales and order of questions can influence the results. Third, researchers should make sure that their implementing partner understands what it will take to conduct a research. Fourth, researchers should make an intentional decision about how, and how much, to incorporate technology into their survey. Fifth, researchers should not assume people will sign up to receive a program or service. They should find out directly whenever possible by piloting or otherwise gauging demand for their intervention.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115476176","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-11-10DOI: 10.23943/princeton/9780691183138.003.0001
Dean S. Karlan, J. Appel
This introductory chapter provides an overview of the growth of randomized controlled trials (RCTs). RCTs found their way into domestic social policy discussions as early as the 1960s when they were used to evaluate government assistance programs, such as negative income tax rates for the poor. In the 1990s, a new crop of development economists began using RCTs in the field to evaluate aid programs. The chapter then argues that researchers must begin talking about failures to ensure evidence plays an appropriate role in policy. The language of RCTs as the “gold standard” for evidence has no doubt helped fuel their rise. However, not every program or theory is amenable to study by an RCT; even when one is, the RCT can be poorly executed, producing no valuable knowledge.
{"title":"Why Failures?","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/princeton/9780691183138.003.0001","DOIUrl":"https://doi.org/10.23943/princeton/9780691183138.003.0001","url":null,"abstract":"This introductory chapter provides an overview of the growth of randomized controlled trials (RCTs). RCTs found their way into domestic social policy discussions as early as the 1960s when they were used to evaluate government assistance programs, such as negative income tax rates for the poor. In the 1990s, a new crop of development economists began using RCTs in the field to evaluate aid programs. The chapter then argues that researchers must begin talking about failures to ensure evidence plays an appropriate role in policy. The language of RCTs as the “gold standard” for evidence has no doubt helped fuel their rise. However, not every program or theory is amenable to study by an RCT; even when one is, the RCT can be poorly executed, producing no valuable knowledge.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"85 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123582971","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-11-10DOI: 10.23943/PRINCETON/9780691183138.003.0006
Dean S. Karlan, J. Appel
This chapter focuses on low participation rates. Low participation rates squeeze the effective sample size for a test, making it more difficult, statistically, to identify a positive treatment effect. There are two moments in which low participation rates can materialize: during the intake process to a study or intervention, or after random assignment to treatment or control. Low participation during the intake process often occurs when marketing a program to the general public. Researchers working in the field with partner organizations often face inflexible constraints in trying to cope with low participation during intake. The second type of low participation—that which occurs after subjects have been randomly assigned to treatment or control—is a more daunting problem and is less likely solvable than low participation at the intake phase.
{"title":"Low Participation Rates","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/PRINCETON/9780691183138.003.0006","DOIUrl":"https://doi.org/10.23943/PRINCETON/9780691183138.003.0006","url":null,"abstract":"This chapter focuses on low participation rates. Low participation rates squeeze the effective sample size for a test, making it more difficult, statistically, to identify a positive treatment effect. There are two moments in which low participation rates can materialize: during the intake process to a study or intervention, or after random assignment to treatment or control. Low participation during the intake process often occurs when marketing a program to the general public. Researchers working in the field with partner organizations often face inflexible constraints in trying to cope with low participation during intake. The second type of low participation—that which occurs after subjects have been randomly assigned to treatment or control—is a more daunting problem and is less likely solvable than low participation at the intake phase.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"73 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133488257","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2017-01-31DOI: 10.23943/PRINCETON/9780691183138.003.0003
Dean S. Karlan, J. Appel
This chapter examines technical design flaws. There are two common issues to avoid when it comes to survey design: bloated surveys, particularly without a clear analysis plan for all questions; and poorly designed survey items. No less important than the survey and other data collection tools is the plan to deploy them. As such, researchers should field test survey questions before launch. Also, debrief regularly with field survey teams to find out which questions respondents are struggling with, which parts of the survey are hardest to administer, and the like. The chapter then considers mistakes in randomization, power, and necessary sample size calculations in RCT design. Power and necessary sample size calculations rely on parameters that are hard to observe or guess. The best advice is to run these calculations multiple times, imagining a range of scenarios in the field and using a corresponding range of values for key parameters.
{"title":"Technical Design Flaws","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/PRINCETON/9780691183138.003.0003","DOIUrl":"https://doi.org/10.23943/PRINCETON/9780691183138.003.0003","url":null,"abstract":"This chapter examines technical design flaws. There are two common issues to avoid when it comes to survey design: bloated surveys, particularly without a clear analysis plan for all questions; and poorly designed survey items. No less important than the survey and other data collection tools is the plan to deploy them. As such, researchers should field test survey questions before launch. Also, debrief regularly with field survey teams to find out which questions respondents are struggling with, which parts of the survey are hardest to administer, and the like. The chapter then considers mistakes in randomization, power, and necessary sample size calculations in RCT design. Power and necessary sample size calculations rely on parameters that are hard to observe or guess. The best advice is to run these calculations multiple times, imagining a range of scenarios in the field and using a corresponding range of values for key parameters.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133752537","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2017-01-31DOI: 10.23943/PRINCETON/9780691183138.003.0004
Dean S. Karlan, J. Appel
This chapter looks at the challenges that arise from working with partner organizations. Evaluations provide evidence about effectiveness and impact, and offer operational insights that can spur improvements to products and processes. From a researcher's perspective, partnerships with practitioners offer access to the people who live and breathe the issues in question. However, along with the mutual benefits of partnerships come unique challenges, one of which is limited staff flexibility and bandwidth. Partner organization staff typically have tremendous expertise, but often experimental protocols require making changes to familiar tasks. This raises the question of staff capacity and appetite to learn new skills and routines. As such, researchers should seek out partners who genuinely want to learn about their programs and products; who are ready, willing, and able to dedicate an appropriate amount of organizational capacity to research; and who are open to the possibility that not all the answers will be positive.
{"title":"Partner Organization Challenges","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/PRINCETON/9780691183138.003.0004","DOIUrl":"https://doi.org/10.23943/PRINCETON/9780691183138.003.0004","url":null,"abstract":"This chapter looks at the challenges that arise from working with partner organizations. Evaluations provide evidence about effectiveness and impact, and offer operational insights that can spur improvements to products and processes. From a researcher's perspective, partnerships with practitioners offer access to the people who live and breathe the issues in question. However, along with the mutual benefits of partnerships come unique challenges, one of which is limited staff flexibility and bandwidth. Partner organization staff typically have tremendous expertise, but often experimental protocols require making changes to familiar tasks. This raises the question of staff capacity and appetite to learn new skills and routines. As such, researchers should seek out partners who genuinely want to learn about their programs and products; who are ready, willing, and able to dedicate an appropriate amount of organizational capacity to research; and who are open to the possibility that not all the answers will be positive.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"35 2","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114026590","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2017-01-31DOI: 10.23943/PRINCETON/9780691183138.003.0002
Dean S. Karlan, J. Appel
This chapter discusses inappropriate research setting. In practice, choosing a setting is often a complex process. It takes time and effort, judgment, and a theory that describes how the underlying context will interact with the treatment to be tested. Problems often arise when researchers try to shoehorn a fit. A few common pitfalls worth mentioning include poorly timed studies; technically infeasible interventions; immature products; and researchers not knowing when to walk away. Ultimately, this is about risk management. As such, researchers must look at physical, social, and political features of the setting. First, these should fit the intervention and the theory that underlies it. Second, context must permit delivery of the intervention. Finally, data collection needs to be possible, whether through survey or administrative data.
{"title":"Inappropriate Research Setting","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/PRINCETON/9780691183138.003.0002","DOIUrl":"https://doi.org/10.23943/PRINCETON/9780691183138.003.0002","url":null,"abstract":"This chapter discusses inappropriate research setting. In practice, choosing a setting is often a complex process. It takes time and effort, judgment, and a theory that describes how the underlying context will interact with the treatment to be tested. Problems often arise when researchers try to shoehorn a fit. A few common pitfalls worth mentioning include poorly timed studies; technically infeasible interventions; immature products; and researchers not knowing when to walk away. Ultimately, this is about risk management. As such, researchers must look at physical, social, and political features of the setting. First, these should fit the intervention and the theory that underlies it. Second, context must permit delivery of the intervention. Finally, data collection needs to be possible, whether through survey or administrative data.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123094762","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2017-01-31DOI: 10.23943/princeton/9780691183138.003.0005
Dean S. Karlan, J. Appel
This chapter assesses survey and measurement execution problems in field research. Until recently, the vast majority of surveys in development field studies were done the old-fashioned way, on clipboards with pen and paper. The past five years have seen a huge shift toward electronic data collection using laptops, personal digital assistants (PDAs), or even smartphones. This has several advantages but also poses risks. It requires electricity to charge devices, often a challenge in rural areas of developing countries. Still, even as laptops, tablets, PDAs, and other technologies are incorporated, surveying remains a very human process. On the upside: surveyors can adapt, interpret, and problem-solve when necessary. On the downside: surveyors can adapt, interpret, and problem-solve whenever they want, which can substantially impact respondents' answers. Meanwhile, some researchers prefer to use measurement tools that capture data directly, without asking questions. The problem with measurement tools is that they do not always work as advertised.
{"title":"Survey and Measurement Execution Problems","authors":"Dean S. Karlan, J. Appel","doi":"10.23943/princeton/9780691183138.003.0005","DOIUrl":"https://doi.org/10.23943/princeton/9780691183138.003.0005","url":null,"abstract":"This chapter assesses survey and measurement execution problems in field research. Until recently, the vast majority of surveys in development field studies were done the old-fashioned way, on clipboards with pen and paper. The past five years have seen a huge shift toward electronic data collection using laptops, personal digital assistants (PDAs), or even smartphones. This has several advantages but also poses risks. It requires electricity to charge devices, often a challenge in rural areas of developing countries. Still, even as laptops, tablets, PDAs, and other technologies are incorporated, surveying remains a very human process. On the upside: surveyors can adapt, interpret, and problem-solve when necessary. On the downside: surveyors can adapt, interpret, and problem-solve whenever they want, which can substantially impact respondents' answers. Meanwhile, some researchers prefer to use measurement tools that capture data directly, without asking questions. The problem with measurement tools is that they do not always work as advertised.","PeriodicalId":340586,"journal":{"name":"Failing in the Field","volume":"86 6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132755975","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：