Neural Networks Using Multiplicative Features Based on Second-Order Statistics for Acoustic and Speech Applications

Abstract

This paper investigates multiplicative interactions such as auto-correlations between features in neural networks. Conventionally, in the field of pattern recognition, including spoken language processing, non-linear relationships among features, e.g., high-order local auto-correlations and multiplicative features seen in sigma-pi cells, have been explored. These features are specifically designed to capture the correlations in the spectro-temporal regions to gain robustness for classification. However, the features based on the multiplicative interactions, or elementary second-order statistics like autocorrelations, have not been well explored in speech processing. Accordingly, there would be open to discussion about the performance improvement of classification problems employing multiplicative features. Thus, we will investigate the multiplicative interactions extracted from spectro-temporal regions through the neural networks. We will conduct the experiments on three kinds of classification tasks, i.e., acoustic event/scene classification and speech recognition, while implementing a simple multiplicative module to produce the interactions between features. Our proposed neural networks with multiplicative blocks achieved promising improvements in all tasks, and the experimental results show that the proposed method improved accuracy by 0.45 % in the acoustic event classification, by 2.15 % in the acoustic scene classification, and the phone error rate (PER) by 6.5 % in the phoneme recognition.