An advanced learning environment and a scalable deep reinforcement learning approach for rolling stock circulation on urban rail transit line

Rolling stock circulation is the process of assigning rolling stocks to a set of predetermined train trips with fixed departure and arrival times. This paper considers mathematical models, solving approaches, and numerical experiments for hypothesized and real-world cases of rolling stock circulation, in which two end-point depots of an urban rail transit line are involved. The objective aims to minimize the total number of rolling stocks in utilization, to balance the workload of the utilized rolling stocks, and to balance the numbers of rolling stocks available at each depot at the beginning and the end of the planning horizon. To achieve the goals, a multi-commodity flow model and a deep reinforcement learning framework for the rolling stock circulation problem are proposed, accommodating the use of multiple types of rolling stocks, of which the former is a non-linear integer programming model. The multi-commodity flow model is solved by the CP Optimizer embedded in ILOG CPLEX and a custom-developed Ant Colony Optimization algorithm, serving as the exact and heuristic benchmarks respectively. The rolling stock circulation problem is innovatively modeled as a Markov decision process within the deep reinforcement learning framework, incorporating an advanced learning environment. This environment is designed by embedding state definition, constraint detection, and reward assignment, enabling effective interaction with the agent. A proximal policy optimization algorithm with a proximal policy update mechanism and adaptive policy-learning rates is adopted to solve the proposed problem. Numerical experiments on hypothesized and real-world cases illustrate the effectiveness of the proposed deep reinforcement learning method for rolling stock circulation. Compared to the benchmark approach, deep reinforcement learning can improve the solution quality with the problem scale increasing, which proves the adaptiveness to applications with complex environments and large state spaces and shows the strong potential to generalize across problems with different scales.