A 109-GOPs/W FPGA-Based Vision Transformer Accelerator With Weight-Loop Dataflow Featuring Data Reusing and Resource Saving

The Vision Transformer (ViT) models have demonstrated excellent performance in computer vision tasks, but a large amount of computation and memory access for massive matrix multiplications lead to degraded hardware performance compared to convolutional neural network (CNN). In this paper, we propose a ViT accelerator with a novel “Weight-Loop” dataflow and its computing unit, for efficient matrix multiplication computation. By data partitioning and rearrangement, the number of memory accesses and the number of registers are greatly reduced, and the adder trees are eliminated. A computation pipeline with the proposed dataflow scheduling method is constructed to maintain a high utilization rate through zero bubble switching. Moreover, a novel accurate dual INT8 multiply-accumulate (DI8MAC) method for DSP optimization is introduced to eliminate the additional correction circuits by weight encoding. Verified in the Xilinx XCZU9EG FPGA, the proposed ViT accelerator achieves the lowest inference latencies of 3.91 ms and 13.98 ms for ViT-S and ViT-B, respectively. The throughput of the accelerator can reach up to 2330.2 GOPs with an energy efficiency of 109 GOPs/W, showing a significant improvement compared to the state-of-the-art works.