Coding theory and regularization

Abstract

This paper uses two principles, the robust encoding of residuals and the efficient coding of parameters, to obtain a new learning rule for neural networks. In particular, it examines how different coding techniques give rise to different learning rules. The storage space requirements of parameters and residuals are considered. A 'group regularizer' is derived from encoding of the parameters as a whole group rather than individually.<>