Chinese coding type identification based on Kolmogorov complexity theory

Abstract

Identification of Chinese coding type is a major and challenging issue in Chinese web content audit and analysis. In this paper we develop a novel algorithm based on the theory of Kolmogorov complexity to identify the coding type of Chinese characters of a given text segment. An array of text compressors are used as filters to evaluate the information distance of text under examination and the training corpus coded in different coding type. The information distance can be used to decide the coding type according to the Kolmogorov theory. In this paper a particular compressing algorithm is used to minimize computing complexity by separating coding book training stage and compressing stage. Finally, we present the experimental results through which the accuracy and performance of the algorithm is confirmed. The result also proves that this algorithm is especially efficient when short text segment is under examination comparing with the n-gram algorithms.