Using Multi-Arm Designs to Test Operating Welfare-to-Work Programs.

Abstract

Background: In the early 1970s, most researchers thought that randomized controlled trials (RCTs) could not be used to measure the effectiveness of large-scale operating welfare reform and employment programs. By the mid-1970s, the Supported Work Demonstration showed that, under certain conditions, this was both feasible and valuable. However, the experimental design was simple; a multi-arm test had been rejected as unrealistic. Within 10 years, a three-arm design was implemented in San Diego to assess both a welfare-to-work program's overall impact and the contribution of a specific component. Less than 10 years later, the Job Opportunities and Basic Skills Training (JOBS)/National Evaluation of Welfare-to-Work Strategies (NEWWS) study used a more complex design to determine the relative effectiveness of two strategies operated in the same locations: one emphasizing getting a job quickly and the other requiring basic education. In San Diego and JOBS/NEWWS, the tested reforms emerged from political processes and were funded through regular program budgets. In both cases, researchers inserted multi-arm RCTs into operating welfare offices, trading control over the treatment for scale (thousands of people) and real-world conditions. Both RCTs were successfully implemented.

Objectives and results: This article examines why multi-arm designs were attempted, how they were structured, why public administrators cooperated, what various actors sought to learn, and how the researchers determined what strategies the different experimental arms ended up to truly represent. The article concludes that these designs provide convincing evidence and can be inserted into operating programs if the studies address questions that are of keen and immediate interest to state or local program administrators and researchers.