Hardware implementation of Multi-Rate input SoftMax activation function

Abstract

The SoftMax activation function is a normalized exponential function that is usually used as an activation function of the last layer of a fully connected neural network. The number of neurons in this layer represents the number of classes. The SoftMax activation function is used to normalize the network outputs to a probability distribution over predicted output classes. In this paper, a multi-rate input SoftMax activation function has been designed and built on FPGA. The unit can read 4 or 2 consecutive inputs or one input, every predefined number of cycles. A ROM design has been utilized to determine the exponential part of the function, while the Coordinate Rotation Digital Computer (CORDIC) reciprocal algorithm has been used to calculate the reciprocal of the sum of the input exponential. Hardware multipliers have been used to calculate the SoftMax output. Unit optimization is achieved by pipelining on the input and output stages. The unit can be configured and controlled by an ARM microcontroller as a complete System-on-Chip (SoC) built on Field Programmable Gate Array (FPGA).