A survey on learning with noisy labels in Natural Language Processing: How to train models with label noise

Abstract

When applying deep neural network language models to related systems (e.g., question answering systems, chatbots, and intelligent assistants), many datasets contain different types or degrees of label noise. Label noise can lead to a decline in model performance and an increase in resource consumption. Therefore, learning with noisy labels is becoming an important task in Natural Language Processing (NLP). This paper aims to collect, analyze, and evaluate methods for learning with label noise in NLP. First, we analyze the relationship between data feature extraction, prediction output, and optimization in the context of noise robustness to help researchers understand the mechanisms behind noise generation. Based on this, we classified the noise processing methods into five types according to the training process: feature vector, transition matrix, prediction confidence, loss improvement, and data weighting. We analyze each method and conduct a systematic evaluation across six metrics. In addition, we summarized the commonly used resources such as datasets, open source codes, etc. Finally, we also analyzed the challenges faced in current research and the potential opportunities. As a comprehensive survey, this work will help researchers and industry developers to understand the current state of research and unique challenges facing label-noise learning, which facilitate the selection and combination of different methods in applications to further advancements.