Accelerating Compact Convolutional Neural Networks with Multi-threaded Data Streaming

Abstract

Recent advances in convolutional neural networks (CNNs) reveal the trend towards designing compact structures such as MobileNet, which adopts variations of traditional computing kernels such as pointwise and depthwise convolution. Such modified operations significantly reduce model size with an only slight degradation in inference accuracy. State-of-the-art neural accelerators have not yet fully exploit algorithmic parallelism for such computing kernels in compact CNNs. In this work, we propose a multithreaded data streaming architecture for fast and highly parallel execution of pointwise and depthwise convolution, which can be also dynamically reconfigured to process conventional convolution, pooling, and fully connected network layers. The architecture achieves efficient memory bandwidth utilization by exploiting two modes of data alignment. We profile MobileNet on the proposed architecture and demonstrate a 9:36x speed-up compared to single-threaded architecture.