DCNN-BiGRU Text Classification Model Based on BERT Embedding

Abstract

Text Classification is a hot topic in natural language processing. In view of the strong correlation the structure of natural language, direct translation the text into vector will lead to too high dimension. In addition, traditional word vector usually maps words with a single vector, which cannot represent the polyseme of words and affect the accuracy of the final classification. In this paper, we propose a novel DCNN-BiGRU (Deep Convolutional Neural Network Bidirection Gated Recurrent) text classification model based on BERT(Bidirectional Encoder Representations from Transformer) embedding. The model adopts the BERT to train the language model of word semantic representation. The semantic vector is generated dynamically according to the context of the word, and then it is put into the DCNN-BiGRU hybrid model. By doing so, the semantic vector not only contains the local features of text but also the context features of text. Experiments on CCERT Chinese email sample set and movie comment data set verify the validity of this model.