Encoding in Neural Networks

Abstract

Data transforms, parameter re-normalization, and activation functions have gained significant attention in the neural network community over the past several years for improving convergence speed. The results in the literature are for computer vision applications, with batch-normalization (BN) and the Rectified Linear Unit (ReLU) activation attracting attention. In this paper, we present a new approach in data transformation in the context of regression during the synthesis of Head-related Transfer Functions (HRTFs) in the field of audio. The encoding technique whitens the real-valued input data delivered to the first hidden layer of a fully-connected neural network (FCNN) thereby providing the training speedup. The experimental results demonstrate, in a statistically significant way, that the presented data encoding approach outperforms other forms of normalization in terms of convergence speed, lower mean-square error, and robustness to network parameter initialization. Towards this, we used some popular first-and second-order gradient techniques such as scaled conjugate gradient, Extreme Learning Machine (ELM), and stochastic gradient descent with momentum and batch normalization. The improvements, as shown through t-SNE based depiction and analysis on the input covariance matrix, confirm the reduction in the condition number of the input covariance matrix (a process similar to whitening).