On the Detection of Performance Regression Introducing Code Changes: Experience from the Git Project

For many software applications, performance is a critical Non-Functional requirement. Different software testing techniques are associated with various types of software testing, often related to performance regressions. Detecting code changes responsible for performance regression, for a rapidly evolving software with an increasing number of daily commits, is becoming arduous due to performance tests being time-consuming. The expense of running performance benchmarks, for all committed changes, has evolved to the bottleneck of detecting performance regression. Therefore, a recent technique called Perphecy was proposed to help, with quickly identifying performance regression introducing code changes, supporting the selection of performance tests, and reducing their execution time. However, Perphecy was not thoroughly tested on a large system, and so, its performance is still unknown in a real-world scenario. In this paper, we perform an in-depth analysis of Perphecy’s ability to identify performance regression introducing code changes on the open-source Git project. Our work challenges the ability of the model to sustain its performance when increasing the sample under test from 201 commits, to 8596 commits. In addition to verifying the scalability of the previous findings, we also test the efficiency of the proposed approach against a wider variety of performance regression introducing code changes. We provide insights into its advantages, limitations, and practical value.