Constructive proof of efficient pattern storage in the multi-layer perceptron

Abstract

We show that the pattern storage capability of the Gabor polynomial is much higher than the commonly used lower bound on multi-layer perceptron (MLP) pattern storage. We also show that multi-layer perceptron networks having second and third degree polynomial activations can be constructed which efficiently implement Gabor polynomials and therefore have the same high pattern storage capability. The polynomial networks can be mapped to conventional sigmoidal MLPs having the same efficiency. It is shown that training techniques like output weight optimization and conjugate gradient attain only the lower bound of pattern storage. Certainly they are not the final solutions to the MLP training problem.<>