Historical facts learning from Long-Short Terms with Language Model for Temporal Knowledge Graph Reasoning

Abstract

Temporal Knowledge Graph Reasoning (TKGR) aims to reason the missing parts in TKGs based on historical facts from different time periods. Traditional GCN-based TKGR models depend on structured relations between entities. To utilize the rich linguistic information in TKGs, some models have focused on applying pre-trained language models (PLMs) to TKGR. However, previous PLM-based models still face some issues: (1) they did not mine the associations in relations; (2) they did not differentiate the impact of historical facts from different time periods. (3) they introduced external knowledge to enhance the performance without fully utilizing the inherent reasoning capabilities of PLMs. To deal with these issues, we propose HFL: Historical Facts Learning from Long-Short Terms with Language Model for TKGR. Firstly, we construct time tokens for different types of time intervals to use timestamps and input the historical facts relevant to the query into the PLMs to learn the associations in relations. Secondly, we take a multi-perspective sampling strategy to learn from different time periods and use the original text information in TKGs or even no text information to learn reasoning abilities without any external knowledge. Finally, we perform HFL on four TKGR benchmarks, and the experiment results demonstrate that HFL has great competitiveness compared to both graph-based and PLM-based models. Additionally, we design a variant that applies HFL to LLMs and evaluate the performance of different LLMs.