A Comparison of Supervised Text Classification and Resampling Techniques for User Feedback in Bahasa Indonesia

Abstract

User feedback is one of the most important sources of information for improving the quality of software products. Our current research focuses on a software product that is often used in many universities, the E- Learning system. To reduce the effort of manually reading all submitted user feedback, building an automatic text classification using various machine learning approaches is a popular solution. However, there is often a challenge of imbalanced data that could jeopardize the ability of the machine to find the pattern and classify feedback correctly. Several techniques ranging from random resampling of data to artificially creating more data (e.g. SMOTE) have already been proposed for handling imbalanced data and show promising results in terms of performance. This paper aims to implement several numerical representations and implementing resampling techniques (to handling imbalanced data), which then are followed by evaluating some popular supervised machine learning classification algorithms, which are the Logistic Regression, Random Forest, Support Vector Machine, Naive Bayes, and Decision Tree. Finally, evaluating performance with and without using resampling techniques by macro-average F1 Scores. The results show generally the implementation of oversampling techniques leads to better performance, except in a few cases where under-sampling techniques perform better.