Optimization on parametric model

Abstract

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with large number of weights consume considerable storage and memory bandwidth. To address this limitation, prun­ing is an effective way to compress neural networks with high accuracy. To address this limitation, we proposed a method for optimization on parametric model. This method contains three steps. First, we train the network like conventional training. Next, we prune the unimportant connections and retrain the network to get the sparse weight matrix. Finally, we use Singularly Valuable Decomposition (SVD) to do further compression on the sparse weight matrix. Our experiments on MNIST dataset show that our method has the ability on reducing the model size by 4 times and the accuracy could still be kept over 90%.