Capacity control in classifiers for pattern recognition

Abstract

Achieving good performance in statistical pattern recognition requires matching the capacity of the classifier to the size of the available training set. A classifier with too many adjustable parameters (large capacity) is likely to learn the training set without difficulty, but be unable to generalize properly to new patterns. If the capacity is too small, even the training set might not be learned without appreciable error. There is thus an intermediate, optimal classifier capacity which guarantees the best expected generalization for the given training set size. The method of structural risk minimization provides a theoretical tool for tuning the capacity of the classifier to this optimal match. It is noted that the capacity can be controlled through a variety of methods involving not only the structure of the classifier itself, but also properties of the input space that can be modified through preprocessing, as well as modifications of the learning algorithm which regularize the search for solutions to the problem of learning the training set. Experiments performed on a benchmark problem of handwritten digit recognition are discussed.<>