Adam Optimized Pruning Method for Few-Shot Learning with GAN Data Augmentation

Abstract

Weight pruning is widely used for model compression and acceleration. In this work, a novel Adam optimization method for few-shot learning with GAN data augmentation is proposed. The first-order Taylor series is implemented to evaluate parameters' importance toward the loss function. And with the given compression ratio, parameters with importance above the threshold are updated by the Adam optimizer with momentum-accelerated weight decay, while others have negative updates as the penalization. After continuous iterations, the model enables to achieve corresponding sparsity ratio, with the influence of the redundant parameters reducing to a low extent. Experiments demonstrate that this method is effective on ResNet with CUB and ISIC-2018 datasets. Note that CUB and ISIC-2018 are datasets about birds and skin deceases, respectively, which represents the generalization of our method on cross-domain classification issues. As a result, the pruned model is able to retain the accuracy with high model sparse ratios. And in some specific compress ratio, like 10× for CUB dataset and 3 × for ISIC-2018 dataset, the pruned model even outperforms the origin model by 3.15% and 1.16%, respectively.