LV2DMOT: Language and Visual Multimodal Feature Learning for Multiobject Tracking

Abstract

Multiobject tracking (MOT) aims to associate objects of the same identity across video frames, with robust similarity measurement being crucial for maintaining tracking performance. However, the current inefficient integration of motion and appearance cues often leads to tracking failures in challenging scenarios, such as occlusions and missed detections. In this article, we introduce LV2DMOT, a tracker that employs a novel paradigm for integrating motion and appearance cues through language and visual multimodal feature learning, thereby generating more distinctive data association similarities. We propose three key techniques: 1) a text-matching task between tracking trajectories and candidate detections. This method uses text encoding of detection geometric information combined with a temporal model, Mamba, to extract temporal motion features of trajectories, enabling more accurate motion similarity calculations; 2) a multimodal, multilevel feature fusion model that integrates motion and appearance features via a cross-modal learning mechanism, resulting in more robust fused similarities; and 3) a learnable temporal attention model for trajectory appearance feature updates, which effectively aggregates historical visual features to improve the representational ability of trajectory appearance features, employing k-medoids for feature selection. Extensive experiments on the MOT17 and MOT20 datasets demonstrate that our method achieves state-of-the-art (SOTA) tracking performance.