Semantic similarity on multimodal data: A comprehensive survey with applications

Abstract

Recently, the revival of the semantic similarity concept has been featured by the rapidly growing artificial intelligence research fueled by advanced deep learning architectures enabling machine intelligence using multimodal data. Thus, semantic similarity in multimodal data has gained substantial attention among researchers. However, the existing surveys on semantic similarity measures are restricted to a single modality, mainly text, which significantly limits the capability to understand the intelligence of real-world application scenarios. This study critically reviews semantic similarity approaches by shortlisting 223 vital articles from the leading databases and digital libraries to offer a comprehensive and systematic literature survey. The notable contribution is to illuminate the evolving landscape of semantic similarity and its crucial role in understanding, interpreting, and extracting meaningful information from multimodal data. Primarily, it highlights the challenges and opportunities inherent in different modalities, emphasizing the significance of advancements in cross-modal and multimodal semantic similarity approaches with potential application scenarios. Finally, the survey concludes by summarizing valuable future research directions. The insights provided in this survey improve the understanding and pave the way for further innovation by guiding researchers in leveraging the strength of semantic similarity for an extensive range of real-world applications.