Urdu News Classification: An Empirical Study Using Machine Learning Techniques

Abstract

Text is a rich source of information and there is unlimited text on the internet. Automatic text classification is a technique to label those text documents with predefined categories. This has various applications including sentiment analysis, spam detection, NLP etc. There is much work done on english text classification but there is a huge gap with Urdu. There isn't any standard algorithm known that outperforms all others. Also it is observed that classifiers usually perform better when the text is preprocessed, but there aren't any standard stemmer, stop word list, tokenizer etc. available for urdu text. Urdu is rich morphologically and it's a challenge to design preprocessing tools for urdu. This research tends to reduce the gap by testing different classification algorithms using different dimensionality reduction combinations on urdu news data set to know which performs better. It also includes designing a stemmer, tokenizer and preparing a stop word list. In this research it was concluded that SVM performed better with the combination of both preprocessing techniques. Fasttext library was also tested for urdu text classification which achieved 95%accuracy and f-score 1 %less than SVM. Another approach used is that topic modeling has been performed using LDA and documents have been weighed as topics. Classification using documents as topics didn't perform well but Random Forest performed better than Naive Bayes and SVM. It's in future work to design a POS tagger that may improve performance of stemmer and to test deep learning methods for urdu text classification.