Accelerating tensor multiplication by exploring hybrid product with hardware and software co-design

Abstract

Tensor multiplication holds a pivotal position in numerous applications. The existing accelerators predominantly rely on inner or outer products for their computational strategies, yet these methodologies encounter obstacles such as excessive storage overhead, underutilization of parallelism, and merging costs. To tackle the challenges, we propose an acceleration technique that integrates a hybrid product approach with a tailored hardware. Our design can accommodate tensor multiplications of various scales, boasting exceptional scalability. First, we employ a hybrid product approach for tensor multiplications, strategically leveraging various methods – including inner, outer, and Hadamard products – to optimize different stages of submatrices computations. Second, we devise a dedicated architecture that seamlessly aligns with hybrid product, leveraging dataflow paradigm to map tensor multiplication efficiently onto the hardware. Third, we design a sliding-window partial reuse FIFO (SWFIFO), alongside a data reorder and scheduling unit to accelerate data retrieval. For general matrix multiplication (GEMM), our design demonstrates an average speedup of $17.62\times$ and 9.47% energy consumption over Nvidia’s V100 GPU. Furthermore, it surpasses Google’s TPU (size of 256 × 256) by an average of $3.76\times$, TPUv2 (size of 128 × 128) by $3.19\times$ and Eyeriss by $3.8\times$. When evaluated on eight neural network models, our design yields a performance boost of $2.89\times$ over TPU and $2.19\times$ over Eyeriss.