Bilingual IT Service Desk Ticket Classification Using Language Model Pre-training Techniques

Abstract

Language model pre-training techniques have been successfully applied to several natural language processing and text-mining tasks. However, existing published studies regarding automatic IT service desk ticket categorization were mostly conducted using the traditional bag-of-words (BoW) model and focused on the tickets that contain only one language. Therefore, this paper presents an examination of applying the state-of-the-art language model pre-training approaches to automatically determine the service category of bilingual IT service desk tickets, particularly for those tickets that contain Thai and/or English texts. Three well-known algorithms, mBERT, ULMFiT, and XLM-R, are investigated in this study using an in-house real-world dataset. Three Ensemble methods with bag-of-words text representation are used as performance evaluation baselines. According to our experimental results, language model pre-training techniques are superior to the BoW-based Ensemble methods for bilingual IT ticket categorization tasks. XLM-R gives the highest overall performance at 87.02% accuracy and 86.96% F1-score on the test dataset, followed by ULMFiT, mBERT and Ensemble methods, respectively