Why error measures are sub-optimal for training neural network pattern classifiers

Abstract

Pattern classifiers that are trained in a supervised fashion are typically trained with an error measure objective function such as mean-squared error (MSE) or cross-entropy (CE). These classifiers can in theory yield Bayesian discrimination, but in practice they often fail to do so. The authors explain why this happens and identify a number of characteristics that the optimal objective function for training classifiers must have. They show that classification figures of merit (CFM/sub mono/) possess these optimal characteristics, whereas error measures such as MSE and CE do not. The arguments are illustrated with a simple example in which a CFM/sub mono/-trained low-order polynomial neural network approximates Bayesian discrimination on a random scalar with the fewest number of training samples and the minimum functional complexity necessary for the task. A comparable MSE-trained net yields significantly worse discrimination on the same task.<>