A Tool for Online Experiment-Driven Adaptation

Abstract

In this paper, we present Online Experiment-Driven Adaptation (OEDA), a tool for performing end-to-end optimization of a target system abstracted as a black-box by combining statistical and optimization methods and providing statistical guarantees along the optimization process. We present the requirements and architecture of OEDA and describe its built-in optimization process that chains together factorial design, Bayesian optimization, and t-test. OEDA allows the user to create reusable abstractions of systems-to-be-optimized and specify, run and observe the execution of end-to-end experiments. For instance, we support data exchange with common tools like Kafka, MQTT and HTTP. We show the benefits of OEDA in a web server application example. OEDA can be a useful vehicle for research in the area of automated experimentation, an emerging challenge where systems are capable of performing experiments (akin to A/B testing) to themselves in order to self-optimize.