ZoomIn: Discovering Failures by Detecting Wrong Assertions

Abstract

Automatic testing, although useful, is still quite ineffective against faults that do not cause crashes or uncaught exceptions. In the majority of the cases automatic tests do not include oracles, and only in some cases they incorporate assertions that encode the observed behavior instead of the intended behavior, that is if the application under test produces a wrong result, the synthesized assertions will encode wrong expectations that match the actual behavior of the application. In this paper we present Zoom In, a technique that extends the fault-revealing capability of test case generation techniques from crash-only faults to faults that require non-trivial oracles to be detected. Zoom In exploits the knowledge encoded in the manual tests written by developers and the similarity between executions to automatically determine an extremely small set of suspicious assertions that are likely wrong and thus worth manual inspection. Early empirical results show that Zoom In has been able to detect 50% of the analyzed non-crashing faults in the Apache Commons Math library requiring the inspection of less than 1.5% of the assertions automatically generated by EvoSuite.