Using Dynamic and Static Techniques to Establish Traceability Links Between Production Code and Test Code on Python Projects: A Replication Study

Abstract

The relationship between test code and production code, that is, test-to-code traceability, plays an essential role in the verification, reliability, and certification of software systems. Prior work on test-to-code traceability focuses mainly on Java. However, as Python allows more flexible testing styles, it is still unknown whether existing traceability approaches work well on Python projects. In order to address this gap in knowledge, this paper evaluates whether existing traceability approaches can accurately identify test-to-code links in Python projects. We collected seven popular Python projects and carried out an exploratory study at both the method and module levels (involving a total of 3198 test cases). On these projects, we evaluated 15 individual traceability techniques along with cross-level information propagation and four combining resolution strategies. The results reveal that the performance of test-to-code traceability approaches on Python has many differences with Java: (1) most of the existing techniques have poor effectiveness for Python; (2) after augmenting with cross-level information, the recall surprisingly drops; and (3) machine learning based combination approach achieves the best recall but the worst precision. These findings shed light on the best traceability approaches for Python projects, and also provide guidelines for researchers and the Python community.