A Sparse Deep Linear Discriminative Analysis using Sparse Evolutionary Training

Abstract

Deep Linear Discriminative Analysis (DeepLDA) is an effective feature learning method that combines LDA with deep neural network. The core of DeepLDA is putting a LDA based loss function on the top of deep neural network, which is constructed by fully-connected layers. Generally speaking, fully-connected layers will lead to a large consumption of computing resource. What’s more, capacity of the deep neural network may too large to fit training data properly when fully-connected layers are used. Thus, performance of DeepLDA may be improved by increasing sparsity of the deep neural network. In this paper, a sparse training strategy is exploited to train DeepLDA. Dense layers in DeepLDA are replaced by a Erdös-Rényi random graph based sparse topology first. Then, sparse evolutionary training (SET) strategy is employed to train DeepLDA. Preliminary experiments show that DeepLDA trained with SET strategy outperforms DeepLDA trained with fully-connected layers on MINST classification task.