Mixed Precision Based Parallel Optimization of Tensor Mathematical Operations on a New-generation Sunway Processor

Abstract

As an important part of high-performance computing (HPC) applications, tensor mathematical operations have a wide and significant impact on application performance. However, due to the unique heterogeneous architecture and software environment of the new-generation Sunway processors, it is critical to utilize the computing capacities of the processor for tensor mathematical operations. The existing research has not fully considered the computing characteristics of tensor mathematical operations and the hardware features of the new-generation Sunway processor. In this paper, we propose an optimization method for tensor mathematical operations on the new-generation Sunway processor. Firstly, an optimization method for elementary functions is proposed, which implements high-performance vector elementary functions with variable precision. Then, an mixed precision optimization method is proposed, which realizes expression computation with variable precision according to precision requirements of users. Finally, a multi-level parallel optimization method is proposed, which realizes asynchronous parallelism of the master core and the slave cores. The experimental results show that, compared with the native implementation, optimized tensor mathematical operations can achieve an average speedup of 112.19× on 64 cores, which exceeds the theoretical speedup.