An Improved Adaptive and Structured Sentence Embedding

Abstract

Recently, attention mechanism has aroused great interest in various fields of Natural Language Processing (NLP). In this paper, we propose a new model for extracting an interpretable sentence embedding by introducing an "Adaptive self-attention". Instead of using a vector, we use a 2-D matrix to represent the embedding and each valid row of the matrix represents a part of sentence. In addition, a length hierarchy mechanism with a unique loss function is applied to adaptively adjust the number of the valid rows of the matrix, which can solve the problem of attention redundancy in short sentences and lack of attention in long sentences. We evaluate our model on text classification tasks: news categorization, review categorization and opinion classification. The results show that our model, compared with other sentence embedding methods, achieve significant improvement in terms of performance when there exists a large amount of data and the length of the data is evenly distributed.