Reinforcement learning based maintenance scheduling of flexible multi-machine manufacturing systems with varying interactive degradation

Abstract

In flexible multi-machine manufacturing systems, variations in product types dynamically influence machine loads, subsequently affecting the degradation processes of the machines. Moreover, the interactive degradation between the upstream and downstream machines, caused by the product quality deviations, changes with the different production routes for the variable product types. These factors, combined with the uncertain production schedules, present significant challenges for effective maintenance scheduling. To address these challenges, the maintenance scheduling problem is modeled as a Hidden-Mode Markov Decision Process (HM-MDP), where product types are treated as hidden modes that influence machine degradation and the subsequent maintenance decisions. The Interactive Degradation-Aware Proximal Policy Optimization (IDAPPO) reinforcement learning framework is introduced, enhancing the PPO algorithm with Graph Neural Networks (GNNs) to capture interactive degradation among machines and Long Short-Term Memory (LSTM) networks to handle temporal variations in production schedules. An entropy-based exploration strategy further manages the uncertainty of production schedules, enabling IDAPPO to adaptively optimize maintenance actions. Extensive experiments on both small-scale (5-machine) and large-scale (24-machine) systems demonstrate significantly reduced system losses and accelerated convergence of IDAPPO compared to the baseline approaches. These results indicate that IDAPPO provides a scalable and adaptive solution for improving the efficiency and reliability of complex manufacturing environments.