A hybrid lightweight transformer architecture based on fuzzy attention prototypes for multivariate time series classification

Abstract

Multivariate time series classification has become a research hotspot owing to its rapid development. Existing methods mainly focus on the feature correlations of time series, ignoring data uncertainty and sample sparsity. To address these challenges, a hybrid lightweight Transformer architecture based on fuzzy attention prototypes named FapFormer is proposed, in which a convolutional spanning Vision Transformer module is built to perform feature extraction and provide inductive bias, incorporating dynamic feature sampling to select the key features adaptively for increasing the training efficiency. A progressive branching convolution (PBC) block and convolutional self-attention (CSA) block are then introduced to extract both local and global features. Furthermore, a feature complementation strategy is implemented to enable the CSA block to specialize in global dependencies, overcoming the local receptive field limitations of the PBC block. Finally, a novel fuzzy attention prototype learning method is proposed to represent class prototypes for data uncertainty, which employs the distances between prototypes and low-dimensional embeddings for classification. Experiments were conducted using both the UEA benchmark dataset and a practical industrial dataset demonstrate that FapFormer outperforms several state-of-the-art methods, achieving improved accuracy and reduced computational complexity, even under conditions of data uncertainty and sample sparsity.