Text classification based on LSTM and attention

Abstract

An improved text classification method combining long short-term memory (LSTM) units and attention mechanism is proposed in this paper. First, the preliminary features are extracted from the convolution layer. Then, LSTM stores context history information with three gate structures - input gates, forget gates, and output gates. Attention mechanism generates semantic code containing the attention probability distribution and highlights the effect of input on the output. This mixed system model optimizes traditional models to represent features more accurately. The simulation shows that the proposed algorithm in this paper outperformed the RNN algorithm and the CNN algorithm which have long-distance dependency problem. Besides, the results also prove that the proposed algorithm works better than the LSTM algorithm by highlighting the impact of critical input in LSTM on the model.