Multimodal deep hierarchical semantic-aligned matrix factorization method for micro-video multi-label classification

Abstract

As one of the typical formats of prevalent user-generated content in social media platforms, micro-videos inherently incorporate multimodal characteristics associated with a group of label concepts. However, existing methods generally explore the consensus features aggregated from all modalities to train a final multi-label predictor, while overlooking fine-grained semantic dependencies between modality and label domains. To address this problem, we present a novel multimodal deep hierarchical semantic-aligned matrix factorization (DHSAMF) method, which is devoted to bridging the dual-domain semantic discrepancies and the inter-modal heterogeneity gap for solving the multi-label classification task of micro-videos. Specifically, we utilize deep matrix factorization to individually explore the hierarchical modality-specific representations. A series of semantic embeddings is introduced to facilitate latent semantic interactions between modality-specific representations and label features in a layerwise manner. To further improve the representation ability of each modality, we leverage underlying correlation structures among instances to adequately mine intra-modal complementary attributes, and maximize the inter-modal alignment by aggregating consensus attributes in an optimal permutation. The experimental results conducted on the MTSVRC and VidOR datasets have demonstrated that our DHSAMF outperforms other state-of-the-art methods by nearly 3% and 4% improvements in terms of the AP metric.