Long Short-Term Memory with Spin-Based Binary and Non-Binary Neurons

Abstract

In this paper, we develop a low-power and area-efficient hardware implementation for Long Short-Term Memory (LSTM) networks as a type of Recurrent Neural Network (RNN). The LSTM network herein employs Resistive Random-Access Memory (ReRAM) based synapses along with spin-based non-binary neurons to achieve energy-efficiency while maintaining comparable accuracy. The proposed neuron provides a novel activation mechanism with five levels of output accuracy to mimic the ideal tanh and sigmoid activation functions. We have examined the performance of an LSTM network for name prediction purposes utilizing ideal, binary, and the proposed non-binary neuron. The comparison of the results shows that our proposed neuron can achieve up to 85% accuracy and perplexity of 1.56, which attains performance similar to algorithmic expectations of near-ideal neurons. The simulations show that our proposed neuron achieves up to 34-fold improvement in energy efficiency and 2-fold area reduction compared to the CMOS-based non-binary designs.