Re-visiting the coupling between mutants and real faults with Defects4J 2.0

Abstract

Mutation analysis is a well-known testing criterion that involves seeding changes in the system under test, i.e. creating mutants, to simulate faults, and measuring the capacity of the test suite to detect these changes. The question of whether real faults are coupled with the mutants is central, as it determines whether tests that detect the mutants will also detect faults that actually occur in code, making the mutants reasonable test requirements. Prior work has explored this question, notably using the Defects4J dataset in Java. As the dataset and the mutation tools used in these prior works have evolved, this work re-visits this question using the newest available versions in order to strengthen and extend prior results. In this work we use 337 real faults from 15 different projects in the Defects4J 2.0.0 dataset, 2,828 test suites, and two well-known Java mutation testing tools (Major and Pitest) to explore (i) to what extent real faults are coupled with mutants, (ii) how both tools compare in terms of producing mutants coupled with faults, (iii) the characteristics of the mutants that are coupled with real faults, and (iv) the characteristics of faults not coupled with the mutants. Most (80.7%) of the faults used were coupled with at least one mutant created by Pitest or Major, most often with mutants created by both tools. All operators used produced a low (<4%) proportion of coupled mutants, although some operators are exclusively coupled to more faults, i.e. coupled to faults where no other operator produces coupled mutants. Finally, faults not coupled with any mutants usually had small fix patches, and although the code related to these faults was mostly affected by the mutation operators used the mutants produces were still not coupled. Results confirm previous findings showing that the coupling effect mostly holds but that additional mutation operators are needed to capture all faults.