Exploiting imbalanced textual and acoustic data for training prosodically-enhanced RNNLMs

Abstract

There have been many attempts in the past to exploit various sources of information in language modelling besides words, for instance prosody or topic information. With neural network based language models, it became easier to make use of this continuous valued information, because the neural network transforms the discrete valued space into a continuous valued space. So far, models incorporating prosodic information were jointly trained on the auxiliary and the textual information from the beginning. However, in practice the auxiliary information is usually only available for a small amount of the training data. In order to fully exploit text and acoustic data, we propose to re-train a recurrent neural network language model, rather than training a language model from scratch. Using this method we achieved perplexity and word error rate reductions for N-best rescoring on the MIT-OCW lecture corpus.