Improving Thai Word Segmentation using HMM: A Case Study of Sentiment Analysis

Abstract

Word segmentation is a basic problem in the natural language processing of non-boundary delimiters language, especially for the Thai language. The ambiguity of the boundaries of the words in the sentence is one of the significant problems that can cause an unknown word and affects the word segmentation accuracy. This paper presents an improving Thai word segmentation using Hidden Markov Model to cope with an unknown word problem. The five-state of left-to-right HMMs are built according to the classes of the unknown word by applied the parts of speech of the Thai language as the observation symbols of the model. To determine the unknown word in the sentence, the String Matching algorithm is first implemented to find overlapping words and unknown words. The unknown words that unidentified by the lexical dictionary are classified according to their classes by the HMMs. Then the word combining rules are applied to determine the proper word boundary and to merge possible characters into words. In addition, the sentiment analysis task of polarity detection was selected as a case study to verify the accuracy of the proposed method. The precision, recall, and F-measure are used for evaluating the efficiency of the proposed method. The empirical results show that both segmented words and polarity classification results obtained by the proposed method tend to outperform the existing methods.