A Transformer-Based Industrial Time Series Prediction Model With Multivariate Dynamic Embedding

Abstract

Industrial time series prediction (ITSP) is critical to the predictive maintenance system of modern industry. However, time-varying conditions and complex industrial processes cause the distribution drift of industrial time series, raising the difficulty of prediction. This article proposes an ITSP model considering distribution information, namely MDEformer. First, the multivariate dynamic embedding (MDE) is designed to provide the property of the channel-binding dynamic distribution awareness. Specifically, a dynamic mode transition and selection module is adopted to exploit dynamic distribution features of time series, and the bidirectional dynamic residual connection integrates dynamic distribution information into embedding vectors to filter distribution change interference. Then, the vanilla Transformer encoder is used to achieve multivariate prediction. Finally, a generative pretraining and fine-tuning strategy is used to enhance the generalization ability in real production scenarios. Extensive results on a real-world zinc smelting dataset illustrate the superiority of MDEformer.