Hypothesis Testing: How to Eliminate Ideas as Soon as Possible

Abstract

Retail Rocket helps web shoppers make better shopping decisions by providing personalized real-time recommendations through multiple channels with over 100MM unique monthly users and 1000+ retail partners. The rapid improvement of the product is important to win on the high-concurrency market of real-time personalization platforms. The necessity of introducing constant innovations and improvements of algorithms for recommendation systems requires correct tools and a process of rapid testing of hypotheses. It's not a secret that 9 out of 10 hypotheses actually do not improve the performance at least. We had the task stated as follows: How to detect and eliminate the idea that doesn't improve as early as possible, to spend a minimum of resources on that process. In the report we will talk about: How we make our process of hypotheses testing faster. One programming language for R&D. Enmity and friendship of offline and online metrics. Why it is difficult to predict the impact of changing diversity of algorithms. What is the benefit of AA/BB online tests. Bayesian statistics for the evaluation of online tests. Roman Zykov is the Chief Data Scientist at the Retail Rocket. In Retail Rocket is responsible for algorithms of personalized and non-personalized recommendations. Previous to Retail Rocket, Roman was the Head of analytics at the biggest e-commerce companies for almost ten years. He received Ms.Sc. in applied mathematics and physics from the MIPhT in 2004.