Dynamic Inspection and Maintenance Scheduling for Multi-State Systems Under Time-Varying Demand: Proximal Policy Optimization

AbstractInspection and maintenance activities are effective ways to reveal and restore the health conditions of many industrial systems, respectively. Most extant works on inspection and maintenance optimization problems assumed that systems operate under a time-invariant demand. Such a simplified assumption is oftentimes violated by a changeable market environment, seasonal factors, and even unexpected emergencies. In this article, with the aim of minimizing the expected total cost associated with inspections, maintenance, and unsupplied demand, a dynamic inspection and maintenance scheduling model is put forth for multi-state systems (MSSs) under a time-varying demand. Non-periodic inspections are performed on the components of an MSS and imperfect maintenance actions are dynamically scheduled based on the inspection results. By introducing the concept of decision epochs, the resulting inspection and maintenance scheduling problem is formulated as a Markov decision process (MDP). The deep reinforcement learning (DRL) method with a proximal policy optimization (PPO) algorithm is customized to cope with the “curse of dimensionality” of the resulting sequential decision problem. As an extra input feature for the agent, the category of decision epochs is formulated to improve the effectiveness of the customized DRL method. A six-component MSS, along with a multi-state coal transportation system, is given to demonstrate the effectiveness of the proposed method.Keywords: multi-state systemdeep reinforcement learningdynamic inspection and maintenance schedulingproximal policy optimizationtime-varying demandDisclaimerAs a service to authors and researchers we are providing this version of an accepted manuscript (AM). Copyediting, typesetting, and review of the resulting proofs will be undertaken on this manuscript before final publication of the Version of Record (VoR). During production and pre-press, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal relate to these versions also. Additional informationNotes on contributorsYiming ChenYiming Chen received the Ph.D. degrees in mechanical engineering from the University of Electronic Science and Technology of China in 2022. He is currently a Lecturer with the College of Marine Equipment and Mechanical Engineering, Jimei University. His research interests include maintenance decisions, stochastic dynamic programming, and deep reinforcement learning.Yu LiuYu Liu is a professor of industrial engineering with the School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China. He received a PhD degree from the University of Electronic Science and Technology of China in 2010. He was a Visiting Predoctoral Fellow with the Department of Mechanical Engineering, Northwestern University, USA, from 2008 to 2010 and a Postdoctoral Research Fellow with the Department of Mechanical Engineering, University of Alberta, Canada, from 2012 to 2013. He has authored or coauthored more than 90 peer reviewed papers in international journals. His research interests include system reliability modeling and analysis, maintenance decisions, prognostics and health management, and design under uncertainty. He is an editorial board member of several international journals, such as Reliability Engineering & System Safety, Quality and Reliability Engineering International and an Associate Editor of IISE Transactions and IEEE Transactions on Reliability.Tangfan XiahouTangfan Xiahou received the M.Sc. and Ph.D. degrees in mechanical engineering from the University of Electronic Science and Technology of China in 2018 and 2022, respectively. He is currently a Lecturer with the School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China. His research interests include reliability modeling under uncertainty, Dempster–Shafer evidence theory, and prognostics and health management.