Adversarially Regularized Tri-Transformer Fusion for continual multimodal egocentric activity recognition

Abstract

Continual egocentric activity recognition aims to understand first-person activity from the multimodal data captured from wearable devices in streaming environments. Existing continual learning (CL) methods hardly acquire discriminative multimodal representations of activity classes from different isolated stages. To address this issue, this paper proposes an Adversarially Regularized Tri-Transformer Fusion (ARTF) model composed of three frozen transformer backbones with dynamic expansion architecture, which enables flexible and progressive multimodal representation fusion in the CL setting. To mitigate the confusion across different stages, we adopt an adversary-based confusion feature generation strategy to augment unknown classes, explicitly simulating out-stage features that closely resemble those within the stage. Then, the discriminative multimodal fusion representations could be learned by joint training on the current and augmented data at different stages. Experiments show that our model significantly outperforms state-of-the-art CL methods for multimodal continual egocentric activity recognition.