An evolution strategies-based reinforcement learning algorithm for multi-objective dynamic parallel machine scheduling problems

Abstract

The multi-objective dynamic parallel machine scheduling (PMS) problem is a complex combinatorial optimization challenge encountered in manufacturing systems. Various uncertainties exist in the real-world dynamic PMS problem, such as job release time, processing time, and flexible preventive maintenance for machines. The goal is simultaneously optimizing multiple objectives under dynamic and uncertain environments, such as makespan, total tardiness, and energy consumption. This paper proposes an evolution strategies-based reinforcement learning (ESRL) algorithm to address the current multi-objective dynamic PMS problem. The proposed algorithm leverages the exploration capabilities of evolution strategies to evolve effective policies for reinforcement learning in dynamic scheduling. Moreover, the efficiency of the ESRL algorithm is enhanced by implanting three features: a) train the policy to iteratively produce the sequence directly and mitigate the sparse reward issue resulting from the symmetry inherent in the given problem; b) a multi-agent system with independent interaction and centralized training to generate the PMS policy simultaneously; c) a non-dominated sorting mechanism to determine fitness function. Extensive computational experimental results show that the ESRL algorithm outperforms the comparison state-of-the-art evolutionary algorithms and priority dispatching rules in terms of solution quality, convergence, and efficiency, with the advantage of the C-matrix exceeding 60 %, and the advantages in GD and NR surpassing 50 %. Furthermore, ablation experiments demonstrate the significant contributions of additional features in ESRL in enhancing the algorithm's performance. Meanwhile, the results of generalization experiments indicate that the ESRL quickly generates Pareto optimal solutions allowing the trained model to make optimal scheduling decisions.