Detecting Coincidental Correctness and Mitigating Its Impacts on Localizing Variability Faults

Coincidental correctness is the phenomenon that test cases execute faulty statements yet still produce correct/expected outputs. In software testing, this problem is prevalent and causes negative impacts on fault localization performance. Although detecting coincidentally correct (CC) tests and mitigating their impacts on localizing faults in non-configurable systems have been studied in-depth, handling CC tests in Software Product Line (SPL) systems have been unexplored. To test an SPL system, products are often sampled, and each product is tested individually. The CC test cases, that occur in the test suite of a product, not only affect the testing results of the corresponding product but also affect the overall testing results of the system. This could negatively affect fault localization performance and decelerate the quality assurance process for the system. In this paper, we introduce DEMiC, a novel approach to detect CC tests and mitigate their impacts on localizing variability faults in SPL systems. Our key idea to detect CC tests is that two similar tests tend to examine similar behaviors of the system and should have a similar testing state (i.e., both passed or failed). If only one of them failed, the other could be coincidentally passed. In addition, we propose several solutions to mitigate the negative impacts of CC tests on variability fault localization at different levels. Our experimental results on +2,6M test cases of five widely used SPL systems show that DEMiC can effectively detect CC tests, with 97% accuracy on average. In addition, DEMiC could help to improve the fault localization performance by 61%.