Iterative filter pruning with combined feature maps and knowledge distillation

Abstract

Convolutional neural networks (CNNs) have been successfully implemented in various computer vision tasks. However, the remarkable achievements are accompanied by high memory and high computation, which hinder the deployment and application of CNNs on resource-constrained mobile devices. Filter pruning is proposed as an effective method to solve the above problems. In this paper, we propose an iterative filter pruning method that combines feature map properties and knowledge distillation. This method can maximize the important feature information (e.g., spatial features) in the feature map by calculating the information capacity and feature relevance of the feature map, and then pruning based on the set criteria. Then, the pruned network learns the complete feature information of the standard CNN architecture in order to quickly and completely recover the lost accuracy before the next pruning operation. The alternating operation of pruning and knowledge distillation can effectively and comprehensively achieve network compression. Experiments on image classification datasets via mainstream CNN architectures indicate the effectiveness of our approach. For example, on CIFAR-10, our method reduces Floating Point Operations (FLOPs) by 71.8% and parameters by 71.0% with an accuracy improvement of 0.24% over the ResNet-110 benchmark. On ImageNet, our method achieves 55.6% reduction in FLOPs and 52.5% reduction in model memory at the cost of losing only 0.17% of Top-5 on ResNet-50.