Optimizing Training using Information Theory-Based Curriculum Learning Factory

We present a new system that can automatically generate input paths (syllabus) for a convolutional neural network to follow through a curriculum learning to improve training performance. Our system utilizes information-theoretic content measures of training samples to form syllabus at training time. We treat every sample as 2D random variable where a data point contained in the sample (such as a pixel) is modelled as an independent and identically distributed random variable (i.i.d) realization. We use several information theory methods to rank and determine when a sample is fed to a network by measuring its pixel composition and its relationship to other samples in the training set. Comparative evaluation of multiple state-of-the-art networks, including, GoogleNet, and VGG, on benchmark datasets demonstrate a syllabus that ranks samples using measures such as Joint Entropy between adjacent samples, can improve learning and significantly reduce the amount of training steps required to achieve desirable training accuracy. We present results that indicate our approach can reduce training loss by as much as a factor of 9 compared to conventional training.