Maximization of the gradient function for efficient neural network training

Abstract

In this paper, a faster supervised algorithm (BPfast) for the neural network training is proposed that maximizes the derivative of sigmoid activation function during back-propagation (BP) training. BP adjusts the weights of neural network with minimizing an error function. Due to the presence of derivative information in the weight update rule, BP goes to ‘premature saturation’ that slows down the training convergence. In the saturation region, the derivative information tends to zero. To overcome the problem, BPfast maximizes the derivative of activation function together with minimizing the error function. BPfast is tested on five real world benchmark problems such as breast cancer, diabetes, heart disease, Australian credit card, and horse. BPfast exhibits faster convergence and good generalization ability over standard BP algorithm.