Text-enhanced knowledge graph representation learning with local structure

Abstract

Knowledge graph representation learning entails transforming entities and relationships within a knowledge graph into vectors to enhance downstream tasks. The rise of pre-trained language models has recently promoted text-based approaches for knowledge graph representation learning. However, these methods often need more structural information on knowledge graphs, prompting the challenge of integrating graph structure knowledge into text-based methodologies. To tackle this issue, we introduce a text-enhanced model with local structure (TEGS) that embeds local graph structure details from the knowledge graph into the text encoder. TEGS integrates k-hop neighbor entity information into the text encoder and employs a decoupled attention mechanism to blend relative position encoding and text semantics. This strategy augments learnable content through graph structure information and mitigates the impact of semantic ambiguity via the decoupled attention mechanism. Experimental findings demonstrate TEGS’s effectiveness at fusing graph structure information, resulting in state-of-the-art performance across three datasets in link prediction tasks. In terms of Hit@1, when compared to the previous text-based models, our model demonstrated improvements of 2.1% on WN18RR, 2.4% on FB15k-237, and 2.7% on the NELL-One dataset. Our code is made publicly available on https://github.com/HubuKG/TEGS.