What Is a Multi-Modal Knowledge Graph: A Survey

Abstract

With the explosive growth of multi-modal information on the Internet, the multi-modal knowledge graph (MMKG) has become an important research topic in knowledge graphs to meet the needs of data management and application. Most research on MMKG has taken image-text data as the research object and used the multi-modal deep learning approach to process multi-modal data. In comparison, the structure of the MMKG is no uniform statement. This paper focuses on MMKG, introduces the related theories of multi-modal knowledge, and analyzes several common ideas about its construction. The survey also explains the structural evolution, proposes mirror node alignment to represent cross-modal knowledge for MMKG, lists some tasks' difficulties, and ultimately gives a sample MMKG for the news scene.