IAR 2.0: An Algorithm for Refining Inconsistent Annotations for Time-Series Data Using Discriminative Classifiers

Abstract

The performance of discriminative machine-learning classifiers, such as neural networks, is limited by training label inconsistencies. Even expert-based annotations can suffer from label inconsistencies, especially in the case of ambiguous phenomena-to-annotate. To address this, we propose a novel algorithm, iterative annotation refinement (IAR) 2.0, for refining inconsistent annotations for time-series data. IAR 2.0 uses a procedure that utilizes discriminative classifiers to iteratively combine original annotations with increasingly accurate posterior estimates of classes present in the data. Unlike most existing label refinement approaches, IAR 2.0 offers a simpler yet effective solution for resolving ambiguities in training labels, working with real label noise on time-series data instead of synthetic label noise on image data. We demonstrate the effectiveness of our algorithm through five distinct classification tasks on two highly distinct data modalities. As a result, we show that the labels produced by IAR 2.0 systematically improve classifier performance compared to using the original labels or a previous state-of-the-art method for label refinement. We also conduct a set of controlled simulations to systematically investigate when IAR 2.0 fails to improve on the original training labels. The simulation results demonstrate that IAR 2.0 improves performance in nearly all tested conditions. We also find that the decrease in performance when IAR 2.0 fails is small compared to the average performance gain when IAR 2.0 succeeds, encouraging the use of IAR 2.0 even when the nature of data is unknown. The code is freely available at https://github.com/SPEECHCOG/IAR_2.