Replicating Experimental Impact Estimates with Nonexperimental Methods in the Context of Control-Group Noncompliance

ABSTRACT A growing literature on within-study comparisons (WSC) examines whether and in what context nonexperimental methods can successfully replicate the results of randomized experiments. WSCs require that the experimental and nonexperimental methods assess the same causal estimand. But experiments that include noncompliance in treatment assignment produce a divergence in the causal estimands measured by standard approaches: the experiment-based estimate of the impact of treatment (the complier average causal effect, CACE) applies only to compliers, while the nonexperimental estimate applies to all subjects receiving treatment, including always-takers. We develop a new replication approach that solves this problem by using nonexperimental methods to produce an estimate that can be compared to the experimental intent-to-treat (ITT) impact estimate rather than the CACE. We demonstrate the applicability of the method in a WSC of the effects of charter schools on student achievement. In our example, some members of the randomized control group crossed over to treatment by enrolling in the charter schools. We show that several nonexperimental methods that incorporate pretreatment measures of the outcome of interest can successfully replicate experimental ITT impact estimates when control-group noncompliance (crossover) occurs—even when treatment effects differ for compliers and always takers.