Fine-Tuning BERT Models for Multiclass Amharic News Document Categorization

Abstract

Bidirectional encoder representation from transformer (BERT) models are increasingly being employed in the development of natural language processing (NLP) systems, predominantly for English and other European languages. However, because of the complexity of the language’s morphology and the scarcity of models and resources, the BERT model is not widely employed for Amharic text processing and other NLP applications. This paper describes the fine-tuning of a pretrained BERT model to classify Amharic news documents into different news labels. We modified and retrained the model using a custom news document dataset separated into seven key categories. We utilized 2181 distinct Amharic news articles, each comprising a title, a summary lead, and a comprehensive main body. An experiment was carried out to assess the performance of the fine-tuned BERT model, which achieved 88% accuracy, 88% precision, 87.61% recall, and 87.59% F1-score, respectively. In addition, we evaluated our fine-tuned model against baseline models such as bag-of-words with MLP, Word2Vec with MLP, and fastText classifier utilizing the identical dataset and preprocessing module. Our model outperformed these baselines by 6.3%, 14%, and 8% in terms of accuracy, respectively. In conclusion, our refined BERT model has demonstrated encouraging outcomes in the categorization of Amharic news documents, surpassing conventional methods. Future research could explore further fine-tuning techniques and larger datasets to enhance performance.