Spectrum-Based Fault Localization: Testing Oracles are No Longer Mandatory

Abstract

Spectrum-based Fault Localization (SBFL) is one of the most popular approaches for locating software faults, and has received much attention because of its simplicity and effectiveness. It utilizes the execution result of each test case (failure or pass) and the corresponding coverage information to evaluate the likelihood of each program entity (e.g., a statement or a predicate) being faulty. Different formulas for computing such likelihood have been proposed based on different intuitions. All existing SBFL techniques have assumed the existence of a testing oracle, that is, a mechanism which can determine whether the execution of a test case fails or passes. However, such an assumption does not always hold. Recently, metamorphic testing has been proposed to alleviate the oracle problem. Thus, it is a natural extension to investigate how it can help SBFL techniques to locate faults even without using a testing oracle. Based on the framework of metamorphic testing, we have developed a novel concept of mice as a counterpart of the slice used in the current SBFL techniques. More precisely, in the absence of a testing oracle, we can determine whether an expected characterization of the program is satisfied. The outcomes of dissatisfaction or satisfaction of an expected characterization are then regarded as the counterparts of failed or passed executions, respectively, when a testing oracle exists. Since our approach does not require the existence of a testing oracle, it significantly enhances the applicability of SBFL techniques. Case studies on three popular SBFL techniques (Tarantula, Ochiai and Jaccard) with 9 applications are reported to demonstrate the use of the proposed fault localization technique.