The accuracy of fault prediction in modified code - statistical model vs. expert estimation

Abstract

Fault prediction models still seem to be more popular in academia than in industry. In industry, expert estimations of fault proneness are the most popular methods of deciding where to focus the fault detection efforts. In this paper, we present a study in which we empirically evaluate the accuracy of fault prediction offered by statistical models as compared to expert estimations. The study is industry based. It involves a large telecommunication system and experts that were involved in the development of this system. Expert estimations are compared to simple prediction models built on another large system, also from the telecommunication domain. We show that the statistical methods clearly outperform the expert estimations. As the main reason for the superiority of the statistical models we see their ability to cope with large datasets, which results in their ability to perform reliable predictions for larger number of components in the system, as well as the ability to perform prediction at a more fine-grain level, e.g., at the class instead of at the component level