Dubious Evidence, Valuable Information.

Abstract

I explore how "low-quality evidence" from program performance might still be useful in decision-making. Conceptually, a local government named "Here" is motivated to consider a program from "Elsewhere" that seems to show year-over-year exemplary performance. Here must manage five sources of uncertainty about whether and how to extrapolate from Elsewhere: chance in assessing Elsewhere's performance; illusion due to confounding variables; estimating the several powers of the program's components; substitutions in the design process made by Elsewhere and contemplated by Here; and estimating whether in the final analysis Here can meet its own breakeven criterion for going ahead. Here can begin with Elsewhere's experience, but it still must do much thinking and information-collecting on its own.