Improved language modelling by unsupervised acquisition of structure

Abstract

The perplexity of corpora is typically reduced by more than 30% compared to advanced n-gram models by a new method for the unsupervised acquisition of structural text models. This method is based on new algorithms for the classification of words and phrases from context and on new sequence finding procedures. These procedures are designed to work fast and accurately on small and large corpora. They are iterated to build a structural model of a corpus. The structural model can be applied to recalculate the scores of a speech recogniser and improves the word accuracy. Further applications such as preprocessing for neural networks and (hidden) Markov models in language processing, which exploit the structure finding capabilities of this model, are proposed.