Evaluating Stream Classifiers with Delayed Labels Information

Abstract

In general, data stream classifiers consider that the actual label of every unlabeled instance is available immediately after it issues a classification. The immediate availability of class labels allows the supervised monitoring of the data distribution and the error rate to verify whether the current classifier is outdated. Further, if a change is detected, the classifier has access to all recent labeled data to update the model. However, this assumption is very optimistic for most (if not all) applications. Given the costs and labor involved to obtain labels, failures in data acquisition or restrictions of the classification problem, a more reasonable assumption would be to consider the delayed availability of class labels. In this paper, we experimentally analyze the impact of latency on the performance of stream classifiers and call the attention of the community for the need to consider this critical variable in the evaluation process. We also make suggestions to avoid possible biased conclusions due to ignoring the delayed nature of stream problems. These are relevant contributions since few studies consider this variable in new algorithms proposals. Also, we propose a new evaluation measure (Kappa-Latency) that takes into account the arrival delay of actual labels to evaluate and compare a set of classifiers.