Overcoming the local-minimum problem in training multilayer perceptrons by gradual deconvexification

Abstract

A method of training neural networks using the risk-averting error (RAE) criterion Jλ (w), which was presented in IJCNN 2001, has the capability to avoid nonglobal local minima, but suffers from a severe limitation on the magnitude of the risk-sensitivity index λ. To eliminating the limitation, an improved method using the normalized RAE (NRAE) Cλ (w) was proposed in ISNN 2012, but it requires a selection of a proper λ, whose range may be dependent on the application. A new training method called the gradual deconvexification (GDC) is proposed in this paper. It starts with a very large λ and gradually decreases it in the training process until a global minimum of Cλ (w) or a good generalization capability is achieved. GDC training method was tested on a large number of numerical examples and produced a very good result in each test.