On the Evaluation of Structured Information Retrieval-Based Bug Localization on 20 C# Projects

Software projects can grow very rapidly, reaching hundreds or thousands of files in a relatively short time span. Therefore, manually finding the source code parts that should be changed in order to fix a failure is a difficult task. Static bug localization techniques provide cost-effective means of finding files related to the failure described in a bug report. Recently, structured information retrieval (IR) has been used to improve the effectiveness of static bug localization, being successfully applied by techniques such as BLUiR, BLUiR+, and AmaLgam. However, there are some significant shortcomings on how these techniques were evaluated. First, virtually all evaluations have been limited to very few projects written in only one object-oriented programming language, particularly Java. Therefore, the effectiveness of these techniques in other widely-used object-oriented languages such as C# is still unknown. Second, the experimental setup for most of the evaluations make simplistic assumptions that do not hold on real-world scenarios, thereby raising doubts about the reported effectiveness of these techniques. In this paper, we evaluate BLUiR, BLUiR+, and AmaLgam on 20 C# projects, providing a first assessment of these techniques on a previously untested object-oriented language. Moreover, we set up an experiment that addresses the simplistic assumptions commonly present in bug localization studies, thereby providing evidence on how their findings may be biased. Finally, we extend the algorithms of existing techniques in order to understand if structured information retrieval can benefit from the use of a wider range of program constructs, including C# constructs inexistent in Java.