A Camera-Based End-to-End Autonomous Driving Framework Combined With Meta-Based Multi-Task Optimization

Abstract

Most existing autonomous driving pipelines can be divided into two broad categories: those based on a modular framework, which can result in error transmission, and those based on an end-to-end framework, which lack interpretability. To overcome these challenges, we propose a novel vision-based multi-task framework that incorporates motion planning, bird’s eye view (BEV) map generation, BEV object prediction, depth estimation, semantic segmentation, and velocity prediction. In particular, we present an improved view transformation module that transforms feature maps into BEV space and predicts future waypoints in BEV space. The multi-task framework can improve performance by sharing information across tasks, and the results of the multi-tasks also improve interpretability. In addition, to address the negative transfer, we introduce intertask affinity, which provides a rough estimate of the relationship between tasks. Moreover, because these relationships may change during training, we use a meta-based multi-task optimization method to dynamically adjust multi-task weighting. We evaluate the performance of our proposed model using the Longest6 and Town05 Long benchmarks of the CARLA simulator. Our model outperforms the current state-of-the-art camera-based models and achieves competitive results with other multimodal methods on both the benchmarks. These results demonstrate the considerable potential of our proposed model for autonomous driving systems. We have also prepared an autonomous driving demonstration using the CARLA simulator which is presented at https://www.youtube.com/watch?v=ctngFH4GSBc.