Spatial-Temporal Multimodal End-to-End Autonomous Driving

Abstract

Autonomous driving requires precise perception of the surrounding environment, and considering the complementarity of sensor data, we propose an end-to-end model of spatial-temporal multimodal fusion using an attention mechanism. Our model uses a fusion of camera and light detection and ranging (LiDAR), which works as follows: (i) The spatial network performs spatial feature learning using images from the range view (RV) representation of LiDAR and red, blue, and green (RGB) images as input, followed by a parallel ResNet18 network for feature extraction and fusion through an attention mechanism; (ii) the temporal network performs the learning of the temporal dimension of spatial features, and uses the current spatial features from the spatial network and the historical spatial features to do attention learning, which enhances the features relevant to the autonomous driving task; (iii) finally, the model uses spatial-temporal features to select a different prediction branch by navigation instructions to perform regression of waypoints. Our model was trained and tested in the CARLA simulator, and experiments showed that it enabled to complete autonomous driving tasks in complex environments, achieving a success rate of 85%, especially with many dynamic objects.