Multi-Object Tracking based on RGB-D Sensors

The accuracy of the multi-object tracking (MOT) based on the 2D camera without depth info is usually poor. In this paper, we propose a MOT method based on sensors composed of the camera and the ultra-wide band (UWB) radar, which are similar to the depth camera (RGB-D camera). First, we establish a backbone network to extract feature maps from video frames captured by a camera. Then, we combine Faster R-CNN with a re-ID branch to detect objects including the category, coordinate and ID. To track objects, we construct a similarity matrix to calculate the data association between the objects and their historical trajectories. The matrix's elements are calculated by the intersection over union (IoU) between the objects and their related two types of trajectories, which are based on the image data and the UWB localization data separately. Finally, the trajectories are updated by the two types of trajectories, and the recognition network is updated by the localization loss. The experimental results show that our method achieves multi-object recognition and tracking, and outperforms previous methods by a large margin on several public datasets.