Automatic Diacritic Recovery with focus on the Quality of the training Corpus for Resource-scarce Languages

Abstract

The development and availability of high quality corpus for many African languages is still hampered by dearth of appropriate software tools and devices. To be able to rapidly create large quantities of high quality corpus of majority of African and Nigerian languages, a diacritic tool is required. The presentation of texts of natural languages without diacritic marks presents significant problems to both human and computational processing systems due to partial or total loss of the accompanying grammatical, syntactic and or semantic information. This paper investigated the effect of diacritic quality of a small-sized training corpus on the classification accuracy of some simple and commonly used machine learning algorithms for diacritic restoration tasks following the character-based approach. The classification accuracy of eight of the diacritic-bearing characters of Yoruba language of Nigeria were investigated. The results show that the completeness and correctness of diacritics has a significant effect on the performance of the algorithms; decision tree algorithm produced the overall best accuracy response of 3.22 % to the data quality improvement. The observations from the learning behaviours of the algorithms suggests that a 100,000 words corpus is adequate to train a decision tree model for automatic diacritic restoration for Yoruba language but insufficient to obtain a state-of-the art results for the LDA, LOGREG and SVM algorithms.