Neural network pruning and hardware acceleration

Neural network pruning is a critical technique to efficiently deploy neural network models on edge devices with limited computing resources. Although many neural network pruning methods have been published, it is difficult to implement such algorithms due to their inherent complexity. In this work, we propose a functional pruning tool for neural network models. Our pruning procedure is simple and easy to be implemented, and efficient for deployment. Our pruning tool automatically detects redundancy inside neural network models and prunes the redundant channels. Doing so reduces the total number of model parameters and hence, compresses the size of the model. This approach significantly reduces the number of FLOPs needed for executing the neural network model and improves the inference runtime. To further improve the inference runtime of the pruned model, we leveraged Apache TVM to deploy the pruned model on the DPU FPGA-based hardware accelerator. To demonstrate our approach, we pruned the VGG-16 model on Flower dataset and reached 53-fold reduction in model size with only 7% drop in validation accuracy. The inference latency is reduced 4-fold on CPU and 16-fold on FPGA for the pruned models, compared with the latency of the base model on CPU.