Named Entity Recognition Based on Pre-training Model and Multi-head Attention Mechanism

Abstract

When processing Chinese named entity recognition, the traditional algorithm model have been having the ambiguity of word segmentation and the singleness of the word vector, and the training consequence of algorithm models was not well. To solve this problem, a BERT-BiLSTM Multi-Attention (PMA-CNER) model was proposed to improve the accuracy of Chinese named entity recognition (CNER). This model used BERT model to embed words based on BiLSTM model, which can extract global context semantic features more effectively. Next, a layer of Multi-head attention mechanism was added behind the BiLSTM layer, which can effectively extract multiple semantic features and overcome the shortage of BiLSTM in obtaining local features. Finally, the experimental results on the CLUSER2020 dataset and the Yudu-S4K dataset show that the accuracy rate is significantly improved, reaching 93.94% and 91.83% respectively.