Subgroup Analysis: Pitfalls, Promise, and Honesty

Abstract

Experiments often focus on recovering an average effect of a treatment on an outcome. A subgroup analysis involves identifying subgroups of observations for which the treatment is particularly efficacious or deleterious. Since these subgroups are not preregistered but instead discovered from the data, significant inferential issues emerge. We discuss methods for conduct honest inference on subgroups, meaning generating valid p -values and confidence intervals which ac-count for the fact that the subgroups were not specified a priori . Central to this approach is the split-sample strategy, where half the data is used to identify ef-fects and the other half to test them. After an intuitive and formal discussion of these issues, we provide simulation evidence and two examples illustrating these concepts in practice.