A bilateral semantic guidance network for detection of off-road freespace with impairments based on joint semantic segmentation and edge detection

Abstract

Freespace detection is one of the key technologies for scene understanding and motion planning in autonomous vehicles. However, current research on freespace detection primarily focuses on obstacles provided by objects above the freespace, such as vehicles, pedestrians, and buildings, while less attention is given to impairments within the freespace, such as potholes, defects, and collapses. Moreover, there is a lack of research on the interpretability of artificial intelligence in freespace detection. In this study, we first construct a large-scale off-road freespace detection dataset with impairments (ORIFD). The dataset comprises a total of 24,000 images representing different weather conditions (day, night, snow, etc.) and terrains (concrete roads, dirt roads, rocky paths, etc.). The impairments include potholes, defects, water puddles, and collapses. In addition to adding semantic labels for freespace and impairments, we also create semantic edge labels to enhance the extraction of scene information. Subsequently, we train a novel semantic guidance network (BSGNet) on this dataset, designed to simultaneously perform freespace detection and semantic edge detection tasks. Our framework consists of a deep extended dual-branch encoder, where one branch aggregates multi-scale semantic features, and the other extracts semantic edge information. We also propose an interactive fusion block (IFB) and a global feature aggregation module (GFAM) to enhance the model's feature representation capabilities. Extensive experiments demonstrate that our model outperforms existing state-of-the-art models, achieving superior performance. Additionally, we employ explainable artificial intelligence (XAI) methods to enhance the trustworthiness of our model and implement a method that combines bird's-eye view with the hybrid A* algorithm for generating effective collision-free paths, further extending the application of our research in autonomous vehicles.