Finding bugs by isolating unit tests

Abstract

Even in simple programs there are hidden assumptions and dependencies between units that are not immediately visible in each involved unit. These dependencies are generally hard to identify and locate, and can lead to subtle faults that are often missed, even by extensive test suites. We propose to leverage existing test suites to identify faults due to hidden dependencies and to identify inadequate test suite design. Rather than just executing entire test suites within frameworks such as JUnit, we execute each test in isolation, thus removing masking effects that might be present in the test suites. We hypothesize that this can reveal previously hidden dependencies between program units or tests. A preliminary study shows that this technique is capable of identifying subtle faults that have lived in a system for 120 revisions, despite failures being reported and despite attempts to fix the fault.