Effective and scalable fault injection using bug reports and generative language models

Abstract

Previous research has shown that artificial faults can be useful in many software engineering tasks such as testing, fault-tolerance assessment, debugging, dependability evaluation, risk analysis, etc. However, such artificial-fault-based applications can be questioned or inaccurate when the considered faults misrepresent real bugs. Since typically, fault injection techniques (i.e. mutation testing) produce a large number of faults by altering ”blindly” the code in arbitrary locations, they are unlikely capable to produce few but relevant real-like faults. In our work, we tackle this challenge by guiding the injection towards resembling bugs that have been previously introduced by developers. For this purpose, we propose iBiR, the first fault injection approach that leverages information from bug reports to inject ”realistic” faults. iBiR injects faults on the locations that are more likely to be related to a given bug-report by applying appropriate inverted fix-patterns, which are manually or automatically crafted by automated-program-repair researchers. We assess our approach using bugs from the Defects4J dataset and show that iBiR outperforms significantly conventional mutation testing in terms of injecting faults that semantically resemble and couple with real ones, in the vast majority of the cases. Similarly, the faults produced by iBiR give significantly better fault-tolerance estimates than conventional mutation testing in around 80% of the cases.