An Evolution Approach for Pre-trained Neural Network Pruning without Original Training Dataset

Abstract

—Model pruning is an important technique in real-world machine learning problems, especially in deep learning. This technique has provided some methods for compressing a large model to a smaller model while retaining the most accuracy. However, a majority of these approaches require a full original training set. This might not always be possible in practice if the model is trained in a large-scale dataset or on a dataset whose release poses privacy. Although we cannot access the original training set in some cases, pre-trained models are available more often. This paper aims to solve the model pruning problem without the initial training set by finding the sub-networks in the initial pre-trained model. We propose an approach of using genetic algorithms (GA) to find the sub-networks systematically and automatically. Experimental results show that our algorithm can find good sub-networks efficiently. Theoretically, if we had unlimited time and hardware power, we could find the optimized sub-networks of any pre-trained model and achieve the best results in the future. Our code and pre-trained models are available at: https://github.com/sun-asterisk-research/ga_pruning_research.