Reliable detection of various objects and road users in the surrounding environment is crucial for the safe operation of automated driving systems (ADS). Despite recent progresses in developing highly accurate object detectors based on Deep Neural Networks (DNNs), they still remain prone to detection errors, which can lead to fatal consequences in safety-critical applications such as ADS. An effective remedy to this problem is to equip the system with run-time monitoring, named as introspection in the context of autonomous systems. Motivated by this, we introduce a novel introspection solution, which operates at the frame level for DNN-based 2D object detection and leverages neural network activation patterns. The proposed approach pre-processes the neural activation patterns of the object detector's backbone using several different modes. To provide extensive comparative analysis and fair comparison, we also adapt and implement several state-of-the-art (SOTA) introspection mechanisms for error detection in 2D object detection, using one-stage and two-stage object detectors evaluated on KITTI and BDD datasets. We compare the performance of the proposed solution in terms of error detection, adaptability to dataset shift, and, computational and memory resource requirements. Our performance evaluation shows that the proposed introspection solution outperforms SOTA methods, achieving an absolute reduction in the missed error ratio of 9% to 17% in the BDD dataset.
{"title":"Run-Time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations","authors":"Hakan Yekta Yatbaz;Mehrdad Dianati;Konstantinos Koufos;Roger Woodman","doi":"10.1109/TIV.2024.3385531","DOIUrl":"https://doi.org/10.1109/TIV.2024.3385531","url":null,"abstract":"Reliable detection of various objects and road users in the surrounding environment is crucial for the safe operation of automated driving systems (ADS). Despite recent progresses in developing highly accurate object detectors based on Deep Neural Networks (DNNs), they still remain prone to detection errors, which can lead to fatal consequences in safety-critical applications such as ADS. An effective remedy to this problem is to equip the system with run-time monitoring, named as \u0000<italic>introspection</i>\u0000 in the context of autonomous systems. Motivated by this, we introduce a novel introspection solution, which operates at the frame level for DNN-based 2D object detection and leverages neural network activation patterns. The proposed approach pre-processes the neural activation patterns of the object detector's backbone using several different modes. To provide extensive comparative analysis and fair comparison, we also adapt and implement several state-of-the-art (SOTA) introspection mechanisms for error detection in 2D object detection, using one-stage and two-stage object detectors evaluated on KITTI and BDD datasets. We compare the performance of the proposed solution in terms of error detection, adaptability to dataset shift, and, computational and memory resource requirements. Our performance evaluation shows that the proposed introspection solution outperforms SOTA methods, achieving an absolute reduction in the missed error ratio of 9% to 17% in the BDD dataset.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 6","pages":"5033-5046"},"PeriodicalIF":14.0,"publicationDate":"2024-04-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141965420","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01DOI: 10.1109/TIV.2024.3413588
{"title":"Share Your Preprint Research with the World!","authors":"","doi":"10.1109/TIV.2024.3413588","DOIUrl":"https://doi.org/10.1109/TIV.2024.3413588","url":null,"abstract":"","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 4","pages":"4820-4820"},"PeriodicalIF":8.2,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10555435","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315166","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
End-to-end motion planners have shown great potential for enabling fully autonomous driving. However, when facing out-of-distribution (OOD) events, these planners might not guarantee the optimal prediction of control commands. To better enhance safety, an end-to-end method that benefits robust and general policy learning from potential OOD events is urgently desirable. In this perspective, Sore4PMP, a hierarchical parallel motion planner, is presented as a suitable solution. Based on raw perception data and descriptive prompts, Sore4PMP can first leverage the advanced generative capabilities of Sora to generate virtual OOD events, and then integrate these events into the decision-making process, thereby enhancing the robustness and generalization of autonomous vehicles (AVs) in emergency scenarios. With a comprehensive outlook, this perspective aims to provide a potential direction for the development of foundation models coupled with autonomous driving and finally promote the safety, efficiency, reliability, and sustainability of AVs.
端到端运动规划器在实现完全自主驾驶方面展现出巨大潜力。然而，当面临失控（OOD）事件时，这些规划器可能无法保证控制指令的最优预测。为了更好地提高安全性，迫切需要一种能从潜在的 OOD 事件中获得稳健而通用的策略学习的端到端方法。从这个角度出发，分层并行运动规划器 Sore4PMP 就是一个合适的解决方案。基于原始感知数据和描述性提示，Sore4PMP 可首先利用 Sora 的高级生成功能生成虚拟 OOD 事件，然后将这些事件整合到决策过程中，从而增强自动驾驶汽车（AV）在紧急情况下的鲁棒性和通用性。本视角以全面的视角，旨在为与自动驾驶相结合的基础模型的发展提供一个潜在的方向，并最终促进自动驾驶汽车的安全性、效率、可靠性和可持续性。
{"title":"Sora for Hierarchical Parallel Motion Planner: A Safe End-to-End Method Against OOD Events","authors":"Siyu Teng;Ran Yan;Xiaotong Zhang;Yuchen Li;Xingxia Wang;Yutong Wang;Yonglin Tian;Hui Yu;Lingxi Li;Long Chen;Fei-Yue Wang","doi":"10.1109/TIV.2024.3392647","DOIUrl":"https://doi.org/10.1109/TIV.2024.3392647","url":null,"abstract":"End-to-end motion planners have shown great potential for enabling fully autonomous driving. However, when facing out-of-distribution (OOD) events, these planners might not guarantee the optimal prediction of control commands. To better enhance safety, an end-to-end method that benefits robust and general policy learning from potential OOD events is urgently desirable. In this perspective, Sore4PMP, a hierarchical parallel motion planner, is presented as a suitable solution. Based on raw perception data and descriptive prompts, Sore4PMP can first leverage the advanced generative capabilities of Sora to generate virtual OOD events, and then integrate these events into the decision-making process, thereby enhancing the robustness and generalization of autonomous vehicles (AVs) in emergency scenarios. With a comprehensive outlook, this perspective aims to provide a potential direction for the development of foundation models coupled with autonomous driving and finally promote the safety, efficiency, reliability, and sustainability of AVs.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 4","pages":"4573-4576"},"PeriodicalIF":8.2,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315146","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01DOI: 10.1109/TIV.2024.3399813
Jingwei Ge;Cheng Chang;Jiawei Zhang;Lingxi Li;Xiaoxiang Na;Yilun Lin;Li Li;Fei-Yue Wang
The deployment of large language models (LLMs) brings challenges to intelligent systems because its capability of integrating large-scale training data facilitates contextual reasoning. This paper envisions a revolution of the LLM based (Artificial) Intelligent Operating Systems (IOS, or AIOS) to support the core of automated vehicles. We explain the structure of this LLM-OS and discuss the resulting benefits and implementation difficulties.
{"title":"LLM-Based Operating Systems for Automated Vehicles: A New Perspective","authors":"Jingwei Ge;Cheng Chang;Jiawei Zhang;Lingxi Li;Xiaoxiang Na;Yilun Lin;Li Li;Fei-Yue Wang","doi":"10.1109/TIV.2024.3399813","DOIUrl":"https://doi.org/10.1109/TIV.2024.3399813","url":null,"abstract":"The deployment of large language models (LLMs) brings challenges to intelligent systems because its capability of integrating large-scale training data facilitates contextual reasoning. This paper envisions a revolution of the LLM based (Artificial) Intelligent Operating Systems (IOS, or AIOS) to support the core of automated vehicles. We explain the structure of this LLM-OS and discuss the resulting benefits and implementation difficulties.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 4","pages":"4563-4567"},"PeriodicalIF":8.2,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315157","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01DOI: 10.1109/TIV.2024.3383599
Iván del Pino;Angel Santamaria-Navarro;Anaís Garrell Zulueta;Fernando Torres;Juan Andrade-Cetto
Terrain analysis is of paramount importance for the safe navigation of autonomous robots. In this study, we introduce GATA, a probabilistic real-time graph-based method for segmentation and traversability analysis of point clouds. In the method, we iteratively refine the parameters of a ground plane model and identify regions imaged by a LiDAR as traversable and non-traversable. The method excels in delivering rapid, high-precision obstacle detection, surpassing existing state-of-the-art methods. Furthermore, our method addresses the need to distinguish between surfaces with varying traversability, such as vegetation or unpaved roads, depending on the specific application. To achieve this, we integrate a shallow neural network, which operates on features extracted from the ground model. This enhancement not only boosts performance but also maintains real-time efficiency, without the need for GPUs. The method is rigorously evaluated using the SemanticKitti dataset and its practicality is showcased through real-world experiments with an urban last-mile delivery autonomous robot.
{"title":"Probabilistic Graph-Based Real-Time Ground Segmentation for Urban Robotics","authors":"Iván del Pino;Angel Santamaria-Navarro;Anaís Garrell Zulueta;Fernando Torres;Juan Andrade-Cetto","doi":"10.1109/TIV.2024.3383599","DOIUrl":"https://doi.org/10.1109/TIV.2024.3383599","url":null,"abstract":"Terrain analysis is of paramount importance for the safe navigation of autonomous robots. In this study, we introduce GATA, a probabilistic real-time graph-based method for segmentation and traversability analysis of point clouds. In the method, we iteratively refine the parameters of a ground plane model and identify regions imaged by a LiDAR as traversable and non-traversable. The method excels in delivering rapid, high-precision obstacle detection, surpassing existing state-of-the-art methods. Furthermore, our method addresses the need to distinguish between surfaces with varying traversability, such as vegetation or unpaved roads, depending on the specific application. To achieve this, we integrate a shallow neural network, which operates on features extracted from the ground model. This enhancement not only boosts performance but also maintains real-time efficiency, without the need for GPUs. The method is rigorously evaluated using the SemanticKitti dataset and its practicality is showcased through real-world experiments with an urban last-mile delivery autonomous robot.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 5","pages":"4989-5002"},"PeriodicalIF":14.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10487036","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141964761","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Autonomous driving based on foundation models has recently garnered widespread attention. However, the risk of hallucinations inherent in foundation models could compromise the safety and reliability of autonomous driving systems. This letter, as part of a series of reports from the Distributed/Decentralized Hybrid Workshop on Foundation/Infrastructure Intelligence (DHW-FII), aims to tackle these issues. We introduce VistaRAG, which integrates retrieval-augmented generation (RAG) technologies into autonomous driving systems based on foundation models, to address the inherent reliability challenges in decision-making. VistaRAG employs a dynamic retrieval mechanism to access highly relevant driving experience, real-time road network status, and other contextual information from external databases. This aids foundation models in informed reasoning and decision-making, thereby enhancing the safety and trustworthiness of foundation-model-based autonomous driving systems under complex traffic scenarios.
{"title":"VistaRAG: Toward Safe and Trustworthy Autonomous Driving Through Retrieval-Augmented Generation","authors":"Xingyuan Dai;Chao Guo;Yun Tang;Haichuan Li;Yutong Wang;Jun Huang;Yonglin Tian;Xin Xia;Yisheng Lv;Fei-Yue Wang","doi":"10.1109/TIV.2024.3396450","DOIUrl":"https://doi.org/10.1109/TIV.2024.3396450","url":null,"abstract":"Autonomous driving based on foundation models has recently garnered widespread attention. However, the risk of hallucinations inherent in foundation models could compromise the safety and reliability of autonomous driving systems. This letter, as part of a series of reports from the Distributed/Decentralized Hybrid Workshop on Foundation/Infrastructure Intelligence (DHW-FII), aims to tackle these issues. We introduce VistaRAG, which integrates retrieval-augmented generation (RAG) technologies into autonomous driving systems based on foundation models, to address the inherent reliability challenges in decision-making. VistaRAG employs a dynamic retrieval mechanism to access highly relevant driving experience, real-time road network status, and other contextual information from external databases. This aids foundation models in informed reasoning and decision-making, thereby enhancing the safety and trustworthiness of foundation-model-based autonomous driving systems under complex traffic scenarios.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 4","pages":"4579-4582"},"PeriodicalIF":8.2,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141315147","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-29DOI: 10.1109/TIV.2024.3382995
Hongchang Chen;Ruiyang Gao;Lili Fan;Erxuan Liu;Wenbo Li;Ruichen Tan;Ying Li;Lei He;Dongpu Cao
The innovative development of intelligent cockpit scenarios and functions brings increasingly enhanced user experiences to drivers and passengers in intelligent vehicles. However, existing research lacks a precise definition of intelligent cockpit scenarios and functions, let alone an understanding of their relationship. In this article, we first define concepts related to scenario and function. Then, we construct the scenario-function system framework. Specifically, the scenarios are divided based on the spatial-temporal dimension, and both scenarios and functions are stratified by their attributes. Finally, the progress and perspectives on scenario understanding are discussed in relation to existing research, especially for emotion and motion sickness recognition.
{"title":"Scenario-Function System for Automotive Intelligent Cockpits: Framework, Research Progress and Perspectives","authors":"Hongchang Chen;Ruiyang Gao;Lili Fan;Erxuan Liu;Wenbo Li;Ruichen Tan;Ying Li;Lei He;Dongpu Cao","doi":"10.1109/TIV.2024.3382995","DOIUrl":"https://doi.org/10.1109/TIV.2024.3382995","url":null,"abstract":"The innovative development of intelligent cockpit scenarios and functions brings increasingly enhanced user experiences to drivers and passengers in intelligent vehicles. However, existing research lacks a precise definition of intelligent cockpit scenarios and functions, let alone an understanding of their relationship. In this article, we first define concepts related to scenario and function. Then, we construct the scenario-function system framework. Specifically, the scenarios are divided based on the spatial-temporal dimension, and both scenarios and functions are stratified by their attributes. Finally, the progress and perspectives on scenario understanding are discussed in relation to existing research, especially for emotion and motion sickness recognition.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 5","pages":"4890-4904"},"PeriodicalIF":14.0,"publicationDate":"2024-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141964756","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-29DOI: 10.1109/TIV.2024.3383108
Ben Fei;Rui Zhang;Weidong Yang;Zhijun Li;Wen-Ming Chen
Point cloud completion concentrates on completing geometric and topological shapes from incomplete 3D shapes. Nevertheless, the unordered nature of point clouds will hamper the generation of high-quality point clouds without predicting structured and topological information of the complete shapes and introducing noisy points. To effectively address the challenges posed by missing topology and noisy points, we introduce SPOFormer, a novel topology-aware model that utilizes surface-projection optimization in a progressive growth manner. SPOFormer consists of three distinct steps for completing the missing topology: (1) Missing Keypoints Prediction. A topology-aware transformer auto-encoder is integrated for missing keypoint prediction. (2) Skeleton Generation. The skeleton generation module produces a new type of representation named skeletons with the aid of keypoints predicted by topology-aware transformer auto-encoder and the partial input. (3) Progressively Growth. We design a progressive growth module to predict final output under Multi-scale Supervision and Surface-projection Optimization . Surface-projection Optimization is firstly devised for point cloud completion, aiming to enforce the generated points to align with the underlying object surface. Experimentally, SPOFormer model achieves an impressive Chamfer Distance- $ell _{1}$ (CD) score of 8.11 on PCN dataset. Furthermore, it attains average CD- $ell _{2}$ scores of 1.13, 1.14, and 1.70 on ShapeNet-55, ShapeNet-34, and ShapeNet-Unseen21 datasets, respectively. Additionally, the model achieves a Maximum Mean Discrepancy (MMD) of 0.523 on the real-world KITTI dataset. These outstanding qualitative and quantitative performances surpass previous approaches by a significant margin, firmly establishing new state-of-the-art performance across various benchmark datasets.
{"title":"Progressive Growth for Point Cloud Completion by Surface-Projection Optimization","authors":"Ben Fei;Rui Zhang;Weidong Yang;Zhijun Li;Wen-Ming Chen","doi":"10.1109/TIV.2024.3383108","DOIUrl":"https://doi.org/10.1109/TIV.2024.3383108","url":null,"abstract":"Point cloud completion concentrates on completing geometric and topological shapes from incomplete 3D shapes. Nevertheless, the unordered nature of point clouds will hamper the generation of high-quality point clouds without predicting structured and topological information of the complete shapes and introducing noisy points. To effectively address the challenges posed by missing topology and noisy points, we introduce SPOFormer, a novel topology-aware model that utilizes surface-projection optimization in a progressive growth manner. SPOFormer consists of three distinct steps for completing the missing topology: (1) \u0000<bold>Missing Keypoints Prediction.</b>\u0000 A topology-aware transformer auto-encoder is integrated for missing keypoint prediction. (2) \u0000<bold>Skeleton Generation.</b>\u0000 The skeleton generation module produces a new type of representation named skeletons with the aid of keypoints predicted by topology-aware transformer auto-encoder and the partial input. (3) \u0000<bold>Progressively Growth.</b>\u0000 We design a progressive growth module to predict final output under \u0000<bold>Multi-scale Supervision</b>\u0000 and \u0000<bold>Surface-projection Optimization</b>\u0000. Surface-projection Optimization is firstly devised for point cloud completion, aiming to enforce the generated points to align with the underlying object surface. Experimentally, SPOFormer model achieves an impressive Chamfer Distance-\u0000<inline-formula><tex-math>$ell _{1}$</tex-math></inline-formula>\u0000 (CD) score of 8.11 on PCN dataset. Furthermore, it attains average CD-\u0000<inline-formula><tex-math>$ell _{2}$</tex-math></inline-formula>\u0000 scores of 1.13, 1.14, and 1.70 on ShapeNet-55, ShapeNet-34, and ShapeNet-Unseen21 datasets, respectively. Additionally, the model achieves a Maximum Mean Discrepancy (MMD) of 0.523 on the real-world KITTI dataset. These outstanding qualitative and quantitative performances surpass previous approaches by a significant margin, firmly establishing new state-of-the-art performance across various benchmark datasets.","PeriodicalId":36532,"journal":{"name":"IEEE Transactions on Intelligent Vehicles","volume":"9 5","pages":"4931-4945"},"PeriodicalIF":14.0,"publicationDate":"2024-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141964750","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：