Adaptive Natural Gradient Method for Learning Neural Networks with Large Data set in Mini-Batch Mode

Abstract

Natural gradient learning, which is one of gradient descent learning methods, is known to have ideal convergence properties in the learning of hierarchical machines such as layered neural networks. However, there are a few limitations that degrades its practical usability: necessity of true probability density function of input variables and heavy computational cost due to matrix inversion. Though its adaptive approximation have been developed, it is basically derived for online learning mode, in which a single update is done for a single data sample. Noting that the on-line learning mode is not appropriate for the tasks with huge number of training data, this paper proposes a practical implementation of natural gradient for mini-batch learning mode, which is the most common setting in the real application with large data set. Computational experiments on benchmark datasets shows the efficiency of the proposed methods.