Digital Twin-Driven VCTS Control: An Iterative Apporach Using Model-Based Reinforcement Learning

Abstract

The Virtually Coupled Train Set (VCTS) is a promising framework to improve the efficiency of urban rail transits (URTs), addressing challenges introduced by time-varying and location-varying passenger flow. However, traditional train control models, relying on liner approximation methods to fit nonlinear dynamic models, cannot meet the safety-critical and latency-sensitive task requirements of VCTS systems. Although artificial intelligence (AI)-based control models are promising, substantial training data and computational resources is challenging in URTs. In response to these challenges, this paper proposes a novel digital twin (DT) driven VCTS framework and an iterative-based control policy learning approach. In the designed DT-driven VCTS system, we collect essential training data from the physical domain, representing the real-world environment. We employ model-based reinforcement learning (MBRL) to learn the dynamic train model and optimize the train control policy in the digital domain, a simulated environment mirroring the physical domain. This approach uniquely leverages model predictions for policy optimization during the training process, adapting to a broad range of scenarios beyond reliance on actual operational data. Furthermore, by employing an iterative learning approach and integrating the physical and digital domains, the train control model and policy can be updated to effectively handle uncertainties and complexities encountered in real-world situations. Extensive experiments validates the effectiveness of our proposed framework, demonstrating its robust performance and adaptability across diverse conditions.