Performance Regression Testing on the Java Virtual Machine Using Statistical Test Oracles

Engineering performance-critical systems often requires manual, expensive fine-tuning of critical application parts such as start-up routines, authentication sequences and transactions. It is highly desirable to protect this investment by regression tests that indicate when performance characteristics such as memory usage or thread allocation change. While traditional testing techniques can be used, they are often too coarse, as systems are tested against static thresholds, and therefore important changes that can result in declining system performance will not be detected. To address these limitations, we propose a novel approach to performance regression testing based on automatically generated statistical test oracles. Machine learning methods are used to detect deviations from the profiles shown in these oracles. We present Buto, a proof-of-concept tool tightly integrated into the JUnit testing framework that can be used to test applications executed on the Java virtual machine (JVM). Buto uses data obtained by transparently monitoring applications through Java Management Extensions (JMX). In this paper we describe the Buto framework and demonstrate how to calibrate the tool using an evaluation based on a set of benchmarking examples.