Heteroscedasticity-aware stratified sampling to improve uplift modeling

Abstract

Randomized controlled trials (RCTs) are conducted in many business applications including online marketing or customer churn prevention to investigate the effect of specific treatments (coupons, retention offers, mailings, etc.). RCTs allow for the estimation of average treatment effects and the training of (uplift) models for the heterogeneity of treatment effects across individuals. The problem with RCTs is that they are costly, and this cost increases with the number of individuals included. These costs have inspired research on how to conduct experiments with a small number of individuals while still obtaining precise treatment effect estimates. We contribute to this literature a heteroskedasticity-aware stratified sampling (HS) scheme. We leverage the fact that different individuals have different noise levels in their outcome and that precise treatment effect estimation requires more observations from the “high-noise” individuals than from the “low-noise” individuals. We show theoretically and empirically that HS sampling yields significantly more precise estimates of the ATE, improves uplift models, and makes their evaluation more reliable compared to RCT data sampled completely randomly. Due to these benefits and the simplicity of our approach, we expect HS sampling to be valuable in many real-world applications in business and beyond.