TK-BERT: Effective Model of Language Representation using Topic-based Knowledge Graphs

Abstract

Recently, the K-BERT model was proposed to add knowledge for language representation in specialized fields. The K-BERT model uses a knowledge graph to perform transfer learning on the pre-trained BERT model. However, the K-BERT model adds the knowledge that exists in the knowledge graph rather than the data relevant to the topic of the input data when using the knowledge graph of the corresponding field. Hence, the K-BERT model can cause confusion in the training. To solve this problem, this study proposes a topic-based knowledge graph BERT (TK-BERT) model, which uses the topic modeling technique. The TK-BERT model divides the knowledge graph by topic using the knowledge graph's topic model and infers the topic for the input sentence, adding only knowledge relevant to the topic. Therefore, the TK-BERT model does not add unnecessary knowledge to the knowledge graph. Moreover, the proposed TK-BERT model outperforms the K-BERT model.