Technical Design Flaws

Abstract

This chapter examines technical design flaws. There are two common issues to avoid when it comes to survey design: bloated surveys, particularly without a clear analysis plan for all questions; and poorly designed survey items. No less important than the survey and other data collection tools is the plan to deploy them. As such, researchers should field test survey questions before launch. Also, debrief regularly with field survey teams to find out which questions respondents are struggling with, which parts of the survey are hardest to administer, and the like. The chapter then considers mistakes in randomization, power, and necessary sample size calculations in RCT design. Power and necessary sample size calculations rely on parameters that are hard to observe or guess. The best advice is to run these calculations multiple times, imagining a range of scenarios in the field and using a corresponding range of values for key parameters.