Supervised learning on large redundant training sets

Abstract

A novel algorithm combining the good properties of offline and online algorithms is introduced. The efficiency of supervised learning algorithms on small-scale problems does not necessarily scale up to large-scale problems. The redundancy of large training sets is reflected as redundancy gradient vectors in the network. Accumulating these gradient vectors implies redundant computations. In order to avoid these redundant computations a learning algorithm has to be able to update weights independently of the size of the training set. The stochastic learning algorithm proposed, the stochastic scaled conjugate gradient (SSCG) algorithm, has this property. Experimentally, it is shown that SSCG converges faster than the online backpropagation algorithm on the nettalk problem.<>