Interval based Weight Initialization Method for Sigmoidal Feedforward Artificial Neural Networks

Abstract

Initial weight choice is an important aspect of the training mechanism for sigmoidal feedforward artificial neural networks. Usually weights are initialized to small random values in the same interval. A proposal is made in the paper to initialize weights such that the input layer to the hidden layer weights are initialized to random values in a manner that weights for distinct hidden nodes belong to distinct intervals. The training algorithm used in the paper is the Resilient Backpropagation algorithm. The efficiency and efficacy of the proposed weight initialization method is demonstrated on 6 function approximation tasks. The obtained results indicate that when the networks are initialized by the proposed method, the networks can reach deeper minimum of the error functional during training, generalize better (have lesser error on data that is not used for training) and are faster in convergence as compared to the usual random weight initialization method.