Prompting Robotic Modalities (PRM): A structured architecture for centralizing language models in complex systems

Abstract

Despite significant advancements in robotics and AI, existing systems often struggle to integrate diverse modalities (e.g., image, sound, actuator data) into a unified framework, resulting in fragmented architectures that limit adaptability, scalability, and explainability. To address these gaps, this paper introduces Prompting Robotic Modalities (PRM), a novel architecture that centralizes language models for controlling and managing complex systems through natural language. In PRM, each system modality (e.g., image, sound, actuator) is handled independently by a Modality Language Model (MLM), while a central Task Modality, powered by a Large Language Model (LLM), orchestrates complex tasks using information from the MLMs. Each MLM is trained on datasets that pair modality-specific data with rich textual descriptions, enabling intuitive, language-based interaction. We validate PRM with two main contributions: (1) ROSGPT_Vision, a new open-source ROS 2 package (available at https://github.com/bilel-bj/ROSGPT_Vision) for visual modality tasks, achieving up to 66% classification accuracy in driver-focus monitoring—surpassing other tested models in its category; and (2) CarMate, a driver-distraction detection application that significantly reduces development time and cost by allowing rapid adaptation to new monitoring tasks via simple prompt adjustments. In addition, we develop a Navigation Language Model (NLM) that converts free-form human language orders into detailed ROS commands, underscoring PRM’s modality-agnostic adaptability. Experimental results demonstrate that PRM simplifies system development, outperforms baseline vision-language approaches in specialized tasks (e.g., driver monitoring), reduces complexity through prompt engineering rather than extensive coding, and enhances explainability via natural-language-based diagnostics. Hence, PRM lays a promising foundation for next-generation complex and robotic systems by integrating advanced language model capabilities at their core, making them more adaptable to new environments, cost-effective, and user-friendly.