Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. To fill the gap, we embark on an initial endeavor with the goal of establishing a unified framework of intelligent multi-modal sensing-communication integration by generalizing the concept of ISAC and providing a comprehensive review under this framework. Inspired by the human synesthesia, the so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such an intelligent integration and details its paradigm for the first time. We commence by justifying the necessity and potential of the new paradigm. Subsequently, we offer a rigorous definition of SoM and zoom into the detailed paradigm, which is summarized as three operational modes realizing the integration. To facilitate SoM research, we overview the prerequisite of SoM research, that is, mixed multi-modal (MMM) datasets, and introduce our work. Built upon the MMM datasets, we introduce the mapping relationships between multi-modal sensing and communications, and discuss how channel modeling can be customized to support the exploration of such relationships. Afterward, aiming at giving a comprehensive survey on the current research status of multi-modal sensing-communication integration, we cover the technological review on SoM-enhance-based and SoM-concert-based applications in transceiver design and environment sensing. To corroborate the rationality and superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design tailored for dynamic scenarios. Finally, we propose some open issues and potential directions to inspire future research efforts on SoM.