Retracted on January 26, 2021

Abstract

Recent years have seen an explosion of domain-specific accelerator for Convolutional Neural Networks (CNN). Most of the prior CNN accelerators target neural networks on image recognition, such as AlexNet, VGG, GoogleNet, ResNet, etc. In this paper, we take a different route and study the acceleration of 3D CNN, which are more computational-intensive than 2D CNN and exhibits more opportunities. After our characterization on representative 3D CNNs, we leverage differential convolution across the temporal dimension, which operates on the temporal delta of imaps for each layer and process the computation bit-serially using only the effectual bits of the temporal delta. To further leverage the spatial locality and temporal locality, and make the architecture general to all CNNs, we propose a control mechanism to dynamically switch across spatial delta dataflow and temporal delta dataflow. We call our design temporal-spatial value aware accelerator (TSVA). Evaluation on a set of representation NN networks shows that TSVA can achieve an average of 4.24x speedup and 1.42x energy efficiency. While we target 3D CNN for video recognition, TSVA could also benefit other general CNNs for continuous batch processing.