Word-Based Bantu Language Identification using Naïve Bayes

Abstract

Language identification of text has become increasingly important as large quantities of text are processed or filtered automatically. It is one of the preprocessing steps in Natural Language Processing (NLP) tasks such as information retrieval and machine translation. Few studies have worked on Bantu Languages in automatic language identification. Language identification is a challenge in Bantu languages because of lack of data and in addition to that, languages which are written similarly like Setswana and Sesotho are also challenging. In this paper, we present a word-based Naïve Bayes classifier to identify words of Sesotho and Setswana language. The classifier was trained with words from both Setswana and Sesotho in a supervised manner. Adjectives, pronouns, adverbs and enumeratives are also included. The classifier shows that the two languages can be individually identified as it gives an accuracy of 71.4%. Despite that when we increase the data to double the number of words, the model increased performance to 78%. We also report that the classifier fails with homographs. The performance could be improved by using more data. Additionally, the syllable identification and sentence identification could be implemented along with word-based classifier.