Scene adaptation in adverse conditions: a multi-sensor fusion framework for roadside traffic perception

Abstract

Robust roadside traffic perception requires integrating the strengths of multi-source sensors under various adverse conditions, which is challenging but indispensable for formulating effective traffic management strategies. One limitation of existing radar-camera perception systems is that they focus on integrating multi-source information without directly considering scene information, leading to difficulties in achieving scene adaptive fusion. How to establish the connection between scene information and multi-source information is the key challenge to solving this problem. In this article, we propose a Scene adaptive Sensor Fusion (SSF) framework that characterizes scene information and integrates it into radar-camera fusion schemes, aiming to achieve high-quality roadside traffic perception. Specifically, we introduce a multi-source object association method that accurately associates multi-source sensor information on the roadside. We then utilize coding techniques to characterize the scene information, including visibility characterization regarding lighting and weather conditions, and road characterization regarding sensor viewpoint. By incorporating sensor and scene information into the fusion model, the SSF framework effectively establishes the connection between them. We evaluate the SSF framework on the Roadside Radar and Video Dataset (RRVD) and the Traffic flow Parameter Estimation Dataset (TPED), both collected from real-world traffic scenarios. Experiments demonstrate that SSF significantly improves vehicle detection accuracy under various adverse conditions compared to traditional single-source sensing methods and other state-of-the-art fusion techniques. Furthermore, vehicle trajectories based on SSF detection results enable accurate traffic parameter estimation, such as volume, speed, and density, in complex and dynamic environments.