Revisiting the ROC curve for diagnostic applications with an unbalanced class distribution

Abstract

This communication investigates the impact on classifier evaluation of a high asymmetry between positive and negatives classes. It points out some necessary precautions when reporting classifier performances using threshold-dependent statistics defined with the confusion matrix. It stresses that, in highly unbalanced datasets, reporting the positive predictive value (PPV) is more appropriate than reporting specificity. More elaborate variables such as F-measure and Matthews' correlation coefficient may also provide a reliable portrait. It further concludes that, in many cases, only a small portion of the receiver operating characteristic (ROC) curve is actually useful, unless very low PPV are judged acceptable. Two remedies are proposed to complement the ROC curve: using a positive tradeoff curve (defined herein) or adding iso-PPV lines (i.e., lines of constant PPV) on the ROC graph. The observations reported in this study contribute to understanding of the impact of asymmetry on classifier performances. They also cast some doubt on the pertinence, when dealing with highly asymmetric problems, of using the area under the ROC curve for threshold-independent assessment of classifier performances.