Resisting Noise in Pseudo Labels: Audible Video Event Parsing With Evidential Learning

Abstract

Perceiving temporal events and discriminating their modality types in audible videos, which is also called audio-visual video parsing (AVVP), is becoming a research hotspot in multimodal video understanding. The AVVP task generally follows weakly supervised learning settings, since only video-level labels are provided. Most existing works usually generate modalitywise pseudo labels (PLs) first and then learn to parse audio or visual events from the audible videos. However, this paradigm inevitably results in two defects: 1) the generated PLs for each modality are not fully reliable, which may confuse models if they are adopted as supervision signals for discriminating modalities; and 2) the absence of temporal annotations increases the ambiguities in localizing foregrounds in videos, furtherly causing models prone to being disturbed by noisy labels. To tackle these problems, we propose a novel AVVP framework termed noise-resistant event parsing (NREP), which introduces evidential deep learning (EDL) to overcome the limitations of noisy pseudo supervision. Specifically, our NREP framework consists of three key components: 1) modalitywise evidential learning (MEL) that discriminates the modality-class dependency; 2) temporalwise evidential learning (TEL) that explores meaningful foregrounds; and 3) foreground-background consistency learning (FBCL) for collaborating two evidential learning branches above. Through perceiving meaningful video content and learning evidence for modality dependencies, our method suppresses the disturbance of noise in generated PLs thus achieving remarkable performance with different PL generation strategies. We evaluate our NREP method on two AVVP benchmark datasets and demonstrate it consistently to establish new state-of-the-art. Our implementation codes are available at https://github.com/CFM-MSG/NREP.