Accelerating the training of feedforward neural networks using generalized Hebbian rules for initializing the internal representations

Abstract

It is argued in this paper that most of the problems associated with the application of existing learning algorithms in complex training tasks can be overcome by using only the input data to determine the role of the hidden units, which form a data compression or a data expansion layer. The initial set of internal representations can be formed through an unsupervised learning process applied before the supervised training algorithm. The synaptic weights that connect the input of the network with the hidden units can be determined through various linear or nonlinear variations of a generalized Hebbian learning rule, known as the Oja's rule. Several experiments indicated that the use of the proposed initialization of the internal representations improves significantly the convergence of various gradient-descent-based algorithms used to perform nontrivial training tasks.<>