Associated Learning Architecture Equipped With Parallel Impedance Sensing Strategy to Enhance Cross-Modal Object Perception

Human beings can infer the shape and material characteristics of grasping objects based on multisensory information, which is still a technical challenge for modern robots. The cross-modal object perception mechanism holds promise to assist robots in effectively executing various operations or interactive tasks in complex applications, particularly in harsh visual scenes. Here, we present an associated learning architecture equipped with a parallel impedance sensing strategy, which enhances the perception of captured objects by integrating visual data with somatosensory data from frequency division multiplexing (FDM) parallel impedance and finger bending angles of the robotic hand. We design a cross-modal generative adversarial network (CGAN) in this architecture to achieve cross-modal feature learning for two types of sensory data, mimicking the psychological cognition of human senses. Additionally, the dynamic attention fusion mechanism is employed for feature transfer and fusion learning, enabling the network to adaptively adjust weights based on input cross-modal features, resulting in dynamic feature fusion. The architecture has undergone training and testing with ten categories of objects, successfully achieving cross-modal feature learning and fusion recognition of the two sensory data. Under low-quality image conditions, the recognition accuracy of attention fusion reaches up to 94.0%, significantly surpassing the accuracy of vision alone. This highlights the potential of our architecture to enhance robots to accurately perceive the outside world by integrating visual and somatosensory data, especially in challenging visual environments.