Are There Any Unit Tests? An Empirical Study on Unit Testing in Open Source Python Projects

Abstract

Unit testing is an essential practice in Extreme Programming (XP) and Test-driven Development (TDD) and used in many software lifecycle models. Additionally, a lot of literature deals with this topic. Therefore, it can be expected that it is widely used among developers. Despite its importance, there is no empirical study which investigates, whether unit tests are used by developers in real life projects at all. This paper presents such a study, where we collected and analyzed data from over 70K revisions of 10 different Python projects. Based on two different definitions of unit testing, we calculated the actual number of unit tests and compared it with the expected number (as inferred from the intentions of the developers), had a look at the mocking behavior of developers, and at the evolution of the number of unit tests. Our main findings show, (i) that developers believe that they are developing more unit tests than they actually do, (ii) most projects have a very small amount of unit tests, (iii) developers make use of mocks, but these do not have a significant influence on the number of unit tests, (iv) four different patterns for the evolution of the number of unit tests could be detected, and (v) the used unit test definition has an influence on the results.