Recent research trends on Model Compression and Knowledge Transfer in CNNs

Convolutional neural network (CNN) is the main tool for deep learning and computer vision, and it has many applications in face recognition, sign language recognition and speech recognition. As deep learning becomes more and more mature, the application of convolutional neural networks will become more and more widespread. As we know, the deeper a neural network is, the higher its memory and computational power overhead. Many neural networks used in medicine, autonomous driving, and language recognition have large model complexity, which makes it difficult to apply these CNNs to people's daily life. Therefore, the development of simple, lightweight and small neural networks has become the focus of researchers nowadays. In this paper, we summarize the development of convolutional neural networks in recent years, as well as various methods for compressing models and migrating data from large models to small ones. In general, the main convolutional neural network compression approaches are: pruning, knowledge distillation, aggregating neurons of different scales, proposing new structures, etc. We start from the structure of neural networks, introduce the major structural changes experienced from the development of convolutional neural networks, and then analyze various pruning, compression and knowledge distillation methods. For specific methods, we run different models and compare the improvements of the new methods with respect to the old ones. We also debugged models on adversarial generative pruning, teacher-student networks, and other compressed CNNs during this period, and drew some constructive conclusions. Finally, we summarize the trends in CNN development in recent years and the challenges we may face in the future.