Experimentation Pitfalls to Avoid in A/B Testing for Online Personalization

Online controlled experiments (also called A/B tests, bucket testing or randomized experiments) have become an habitual practice in numerous companies for measuring the impact of new features and changes deployed to softwares products. In theory, these experiments are one of the simplest methods to evaluate the potential effects that new features have on user's behavior. In practice, however, there are many pitfalls that can obscure the interpretation of results or induce invalid conclusions. There is, in the literature, no shortage of prior work on online controlled experiments addressing these pitfalls and conclusions misinterpretations, but the topic is not tackled considering the specific case of testing personalization features. In this paper, we present some of the experimentation pitfalls that are particularly important for personalization features. To better illustrate each pitfall, we include a combination of theoretical argumentation as well as examples from real company's experiments. While there is clearly value in evaluating personalized features by means of online controlled experiments, there are some pitfalls to bear in mind while testing. With this paper, we aim to increase the experimenters' awareness of leading to improved quality and reliability of the results.