Efficient Hardware Architecture of Softmax Layer in Deep Neural Network

Abstract

Deep neural network (DNN), as a very important machine learning technique in classification and detection tasks for images, video, speech as wellas audio, has recently received tremendous attention. Integral Stochastic Computation (Integral SC), on the other hand, has proved its extraordinary ability in hardware implementation of DNNs. Thesoftmax layer is generally used in multi-classification tasks as a very basic and important network layer in DNNs. However, the hardware implementation of softmax layer is expensive cause the exponentiation and division computation. In this paper, we designed an efficient way to simulate softmax layer in DNNs based on Integral stochastic computing, filling the vacancy of previous academic works. Compared to conventional softmax hardware implementation, our method achieves reduction in power and area by 68% and 41%, respectively.