An Accurate and Compact Hyperbolic Tangent and Sigmoid Computation Based Stochastic Logic

Abstract

In this paper, a proof-of-concept implementation of hyperbolic tanh(ax) and sigmoid(2ax) functions for high-precision as well as compact computational hardware based on stochastic logic is presented. Nonlinear activation introducing the non-linearity in the learning process is one of the critical building blocks of artificial neural networks. Hyperbolic tangent and sigmoid are the most commonly used nonlinear activation functions in machine-learning system such as neural networks. This work demonstrates the stochastic computation of tanh(ax) and sigmoid(2ax) functions-based Bernstein polynomial using a bipolar format. The format conversion from bipolar to unipolar format is involved in our implementation. One achievement is that our proposed implementation is more accurate than the state-of-the-arts including FSM based method, JK-FF and general unipolar division. On average, 90% of improvement of this work in terms of mean square error (MAE) has been achieved while the hardware cost and power consumption are comparable to the previous approaches.