Research on Chinese Text Semantic Matching Base on BERT and BiLSTM_Attention

Abstract

In this paper, to further improve the matching accuracy of text semantic matching, which is based on BERT (Bidirectional Encoder Representations from Transformers) fine-tuning, we propose a novel model, which is combine BERT and Attention-Based Bidirectional Long Short-Term Memory Networks (BERT+BiLSTM_Attention). Indeed, outputs of the BERT are further given as inputs of the Bidirectional Long Short-Term Memory Networks (BiLSTM), after that the attention mechanism is further used to capture the needed interactive information and attend those informative words that have a significant impact on semantics. In order to optimize the model, the piecewise constant decay strategy is adopted to control the learning rate. In addition, the weights of BERT are updated to make our model more suitable to the downstream tasks. Finally, experimental results demonstrate that the accuracy of the proposed model is better than LSTM-DSSM model 7.1% in a large-scale Chinese question matching corpus. And 0.12 points higher than BERT based direct fine-tuning.