Long Text Classification Using Pre-trained Language Model for a Low-Resource Language

Abstract

Text classification is an essential task of Natural Language Processing (NLP) that intends to classify texts into predefined classes. Most recent studies show that transformer-based pre-trained language models such as BERT and RoBERTa have achieved state-of-the-art performance in several downstream NLP tasks. Despite their advantages, these models suffer from one primary drawback of the restricted input size. Because of this limitation, they cannot operate the entire input long texts. This paper presents an approach that utilizes the self-attention mechanism to address the bottleneck of most pre-trained language models of long input texts in the case of Amharic, regarded as a low-resourced language. Specifically, our method carefully investigates the significance of each word in the dataset using a self-attention mechanism. Then identify and select the most relevant words according to their attention scores. Finally, we train our model on the filtered text. Our results show that the approach achieves better performance in terms of accuracy compared to the baseline model.