MUTUAL: Towards Holistic Sensing and Inference in the Operating Room.

Abstract

Embodied AI (E-AI) in the form of intelligent surgical robotics and other agents is calling for data platforms to facilitate its development and deployment. In this work, we present a cross-platform multimodal data recording and streaming software, MUTUAL, successfully deployed on two clinical studies, along with its ROS 2 distributed adaptation, MUTUAL-ROS 2. We describe and compare the two implementations of MUTUAL through their recording performance under different settings. MUTUAL offers robust recording performance at target configurations for multiple modalities, including video, audio, and live expert commentary. While this recording performance is not matched by MUTUAL-ROS 2, we demonstrate its advantages related to real-time streaming capabilities for AI inference and more horizontal scalability, key aspects for E-AI systems in the operating room. Our findings demonstrate that the baseline MUTUAL is well-suited for data curation and offline analysis, whereas MUTUAL-ROS 2, should match the recording reliability of the baseline system under a fully distributed manner where modalities are handled independently by edge computing devices. These insights are critical for advancing the integration of E-AI in surgical practice, ensuring that data infrastructure can support both robust recording and real-time processing needs.