Automated Machine Learning in the Wild

Machine Learning research is progressing at an ever increasing pace. Fueled by technology advances commonly referred to as "Big Data", all data related fields are teaming with scientific and applied activity: our communities explore new application areas, develop new learning algorithms, and continuously scale and improve optimization and estimation methods. But from an industry perspective, many of the most impeding challenges are entirely elsewhere. This talk takes a fresh look at the practical state of affairs in the context of running a large-scale automated machine learning system that supports 50 Billion decision daily on behalf of hundreds of digital advertisers. Some of the key lessons are 1) robustness beats peak performance almost always, 2) support for the constant dynamic fluctuations in the data stream is essential, 3) models exploiting unknowingly any weakness of your metrics, and finally 4) the fact that despite big data, the data you really want never exists.