XLM for Autonomous Driving Systems: A Comprehensive Review

Abstract

Large Language Models (LLMs) have showcased remarkable proficiency in various information-processing tasks. These tasks span from extracting data and summarizing literature to generating content, predictive modeling, decision-making, and system controls. Moreover, Vision Large Models (VLMs) and Multimodal LLMs (MLLMs), which represent the next generation of language models, a.k.a., XLMs, can combine and integrate many data modalities with the strength of language understanding, thus advancing several information-based systems, such as Autonomous Driving Systems (ADS). Indeed, by combining language communication with multimodal sensory inputs, e.g., panoramic images and LiDAR or radar data, accurate driving actions can be taken. In this context, we provide in this survey paper a comprehensive overview of the potential of XLMs towards achieving autonomous driving. Specifically, we review the relevant literature on ADS and XLMs, including their architectures, tools, and frameworks. Then, we detail the proposed approaches to deploy XLMs for autonomous driving solutions. Finally, we provide the related challenges to XLM deployment for ADS and point to future research directions aiming to enable XLM adoption in future ADS frameworks.