Bridging the Gap: A Systematic Benchmarking of Uplift Modeling and Heterogeneous Treatment Effects Methods

Abstract

Choosing the correct method to predict the incremental effect of a treatment on customer response is critical to optimize targeting policies in many important applications such as churn management and patient care. Two research streams, uplift modeling and heterogeneous treatment effects (HTE), have emerged that scrutinize the incremental effect of a treatment on customer response. So far, these research streams mostly remain independent, with few studies comparing methods across these communities. However, if the goal is to estimate the incremental effect in the best possible way or to make a new contribution in the context of targeting policies, ignoring either uplift modeling or HTE methods is a serious omission. To fill this research gap, the authors benchmark 15 methods from both literatures on synthetic and real-world data sets. They perform benchmarking to contrast the performance of different methods from both research streams and to highlight the importance of evaluating methods from uplift modeling and HTE. The results show that although most methods suffer from volatility, some methods perform better and are more robust than others. In addition, the authors demonstrate that using the incremental effect can substantially improve a targeting policy, but only if academics and practitioners evaluate various methods from both uplift modeling and HTE.