A Static Dictionary-Based Approach To Compressing Short Texts

Abstract

In this study, Static Dictionary Compression (SDC) method, which is an approach developed to compress short texts, is proposed. The word-based static dictionaries used in this approach were obtained from clusters formed as a result of running a clustering method repeatedly until certain criteria are met. Short text is compressed with the dictionary that has the largest number of words in common with it. It has been shown by tests conducted with datasets containing short texts in 6 different languages that the proposed method compresses better than the general purpose compression methods Gzip, Bzip2, Zstd and PPMd. In the tests made with the data set containing only English short texts, it has been shown that the SDC method can compress better than the smza, shoco and b64pack methods used to compress short texts, and Brotli, which gives good results in short texts because it uses a static dictionary.