Self-supervised temporal adaptive learning for weakly-supervised temporal action localization

Weakly-supervised temporal action localization (WTAL) identifies and localizes actions in untrimmed videos with only video-level labels. Most methods prioritize discriminative snippets, often neglecting of hard action snippets while focusing on class-specific background. Although recent methods have tackled this issue through temporal modeling, they overlook the local temporal structure of actions. To model such temporal structure effectively, we propose a novel self-supervised temporal adaptive learning (STAL) framework, which is composed of two core parts, i.e. self-supervised temporal learning (STL) network and the adaptive learning unit (ALU). Specifically, STL constructs a self-supervised task by performing an erasure and reconstruction process. This pseudo-label-based method relies on a classification task to perceive continuous temporal information for action localization task. To avoid the disturbance of un-confident pseudo labels during self-supervised learning process, two adaptive learning strategies of ALU are designed from two perspectives. In detail, a task-adaptive learning strategy is used to train the proposed tasks to the best for more reliable pseudo labels. Meanwhile, a score-adaptive learning strategy is designed to balance class activation and attention scores. Experiments on two classical datasets, namely, THUMOS14 and ActivityNet datasets, verify the effectiveness of our method.