DAAR: Dual attention cooperative adaptive pruning rate by data-driven for filter pruning

Abstract

Model compression can address the limitations of deep learning in resource-constrained situations by reducing the computational and storage requirements of the model. Structured pruning has emerged as an important compression technique because of its operational flexibility and effectiveness. However, the existing structural pruning methods have two limitations: 1) They use a single measurement to identify the importance of the filters in all the layers, resulting in a loss of spatial information in the shallow layers. 2) The pruning rate is highly dependent on manual interference, which is highly subjective. In this paper, a filter pruning method called dual attention cooperative adaptive pruning rate (DAAR) is proposed. Specifically, a dual attention module that combines spatial attention and channel attention is proposed to measure the effectiveness of the filters. Spatial attention is used in the shallow layers, and channel attention is used in the deep layers. This allows the filter measurements to consider spatial information effectively. An adaptive pruning rate adjustment strategy is also used to eliminate manual subjectivity, achieving precision pruning of each convolutional layer. The experimental results on various datasets and networks demonstrate that the DAAR method achieves improved model performance after pruning. For example, in the CIFAR10 dataset, the precision increases from 93.5% to 93.75% after removing the floating point operations (FLOPs) of 84.1%, outperforming the state-of-the-art pruning methods.