Temporal multi-modal knowledge graph generation for link prediction.

Abstract

Temporal Multi-Modal Knowledge Graphs (TMMKGs) can be regarded as a synthesis of Temporal Knowledge Graphs (TKGs) and Multi-Modal Knowledge Graphs (MMKGs), combining the characteristics of both. TMMKGs can effectively model dynamic real-world phenomena, particularly in scenarios involving multiple heterogeneous information sources and time series characteristics, such as e-commerce websites, scene recording data, and intelligent transportation systems. We propose a Temporal Multi-Modal Knowledge Graph Generation (TMMKGG) method that can automatically construct TMMKGs, aiming to reduce construction costs. To support this, we construct a dynamic Visual-Audio-Language Multimodal (VALM) dataset, which is particularly suitable for extracting structured knowledge in response to temporal multimodal perception data. TMMKGG explores temporal dynamics and cross-modal integration, enabling multimodal data processing for dynamic knowledge graph generation and utilizing alignment strategies to enhance scene perception. To validate the effectiveness of TMMKGG, we compare it with state-of-the-art dynamic graph generation methods using the VALM dataset. Furthermore, TMMKG exhibits a significant disparity in the ratio of newly introduced entities to their associated newly introduced edges compared to TKGs. Based on this phenomenon, we introduce a Temporal Multi-Modal Link Prediction (TMMLP) method, which outperforms existing state-of-the-art techniques.