QSMT-net: A query-sensitive proposal and multi-temporal-span matching network for video grounding

Abstract

The video grounding task aims to retrieve moments from the videos corresponding to a given textual query. This task poses significant challenges because of the need to comprehend the semantic content of both videos and sentences as well as manage the matching relationship between modalities. Existing approaches struggle to effectively meet this challenge, as they often lack consideration for the diversity in constructing proposals to fit segments from varied scenes and disregard the multi-temporal scale matching relationship between queries and proposals. In this paper, we propose the Query-Sensitive Proposal and Multi-Temporal-Span Matching Network (QSMT-Net), an innovative framework designed to generate more distinctive proposals and to enhance the matching between queries and candidate proposals over varying temporal spans. First, we fortify the connection between modes by instituting fine-grained interactions between video clips and textual words. Subsequently, through a learnable pooling mechanism, we dynamically construct candidate proposals tailored to specific queries, thus implementing a query-sensitive proposal generation strategy. Second, we enhanced the model's ability to differentiate adjacent candidate proposals through the multi-temporal-span matching network, which facilitated selecting the most accurate proposal results under various time scales. Experiments on three widely used benchmarks, Charades-STA, TACoS and ActivityNet Captions, our approach demonstrated significant improvements over state-of-the-art methods, indicating promising advancements in video grounding.