SAUST: A Scheme for Acceleration of Unstructured Sparse Transformer

Abstract

Transformer achieves impressive results on many AI tasks. However, it also introduces a huge amount of computation. Pruning is a promising method to reduce the computation load by generating sparse transformer models. To avoid load imbalance caused by computing involved in zero elements, previous works explore structured pruning combined with hardware acceleration. However, tight constraints in structured pruning usually make training much harder and reach a lower sparsity level in the end. This paper proposes SAUST, a scheme that exploits the high sparsity level of unstructured pruning and addresses the load imbalance problem using both hardware and software methods. FPGA implementation shows that SAUST can achieve 3.35x and 2.76x execution time speedup compared to two state-of-the-art references on hardware accelerators.