An Evaluation of Effort-Aware Fine-Grained Just-in-Time Defect Prediction Methods

CONTEXT: Software defect prediction (SDP) is an active research topic to support software quality assurance (SQA) activities. It was observed that unsupervised prediction models were often competitive with supervised ones at release-level and change-level defect prediction. Fine-grained just-in-time defect prediction focuses on defective files in a change, rather than the whole change. A recent study showed that the fine-grained just-in-time defect prediction was cost-effective in terms of effort-aware performance measures. Those studies did not explore the effectiveness of supervised and unsupervised models at that finer level in terms of effort-aware performance measures. OBJECTIVE: To examine the performance of supervised and unsupervised prediction models in the context of fine-grained defect prediction in terms of effort-aware performance measures. METHOD: Experiments with a time-sensitive approach were conducted to evaluate the predictive performance of supervised and unsupervised methods proposed in past studies. Datasets from OSS projects with manually validated defect links were employed from a past study. RESULTS: The use of manually validated links led to low-performance results. No clear difference among supervised and unsupervised methods was found while CBS+, a supervised method, was the best method in terms of F-measure. Even CBS+ did not achieve reasonable performance. A non-linear learning algorithm did not help the performance improvement. CONCLUSION: No clear preference among unsupervised and supervised methods. CBS+ was the best method on average. The predictive performance was still a challenge.