Towards Reproducible Evaluation of Large-Scale Distributed Systems

Reproducing experimental results is nowadays seen as one of the greatest impairments for the progress of science in general and distributed systems in particular. This stems from the increasing complexity of the systems under study and the inherent complexity of capturing and controlling all variables that can potentially affect experimental results. We argue that this can only be addressed with a systematic approach to all the stages of the evaluation process. This raises the following challenges: i) precisely describe the environment and variables affecting the experiment, ii) minimize the number of (uncontrollable) variables affecting the experiment and iii) have the ability to subject the system under evaluation to controlled fault patterns. In the following, we highlight the research directions we are currently pursuing to address these goals. Our overarching goal is to build an open-source evaluation platform, Angainor, able to deploy an experiment, control the network topology, inject faults, monitor the whole experiment and automatically derive summary statistics of the experimental data.