Impact Evaluations in Data Poor Settings: The Case of Stress-Tolerant Rice Varieties in Bangladesh

Abstract

Impact evaluations of new technologies are critical to assessing and improving investment in national and international development goals. Yet many of these technologies are introduced and promoted at times and in places that lack the necessary data to conduct a strongly identified impact evaluation. We present a new method that combines remotely sensed Earth observation (EO) data, recent advances in machine learning, and socioeconomic survey data so as to allow researchers to conduct impact evaluations of a certain class of technologies when traditional economic data is missing. To demonstrate our approach, we study stress tolerant rice varieties (STRVs) that were introduced in Bangladesh more than a decade ago. Using 20 years of EO data on rice production and flooding, we fail to replicate existing RCT and field trial evidence of STRV effectiveness. We validate this failure to replicate with administrative and household panel data as well as conduct Monte Carlo simulations to test the sensitivity to mismeasurement of past evidence on the effectiveness of STRVs. Our findings speak to conducting large scale, long-term impact evaluations to verify external validity of small scale experimental data while also laying out a path for researchers to conduct similar evaluations in other data poor settings.