Data-Model Hybrid-Driven Safe Reinforcement Learning for Adaptive Avoidance Control Against Unsafe Moving Zones

Abstract

With the gradual application of reinforcement learning (RL), safety has emerged as a paramount concern. This article presents a novel data-model hybrid-driven safe RL (SRL) scheme to address the challenge of avoidance control in the operation domain containing multiple moving unsafe zones. First, the avoidance problem is transformed into the optimal control problem of an augmented system by encoding a barrier function (BF) term into the cost function. Then, using the idea of integral RL (IRL), an adaptive learning algorithm is proposed for generating safe control policies, in which the actor-critic neural network (NN) structure is established with the aid of state-following (StaF) kernel function. The policy iteration process is executed by this structure; specifically, the critic network undergoes gradient-descent adaptation, while the actor network employs gradient projection updating. Particularly, via a state extrapolation technique, both real-time experience and simulated experience are utilized in the learning process. Next, closed-loop stability and weight convergence are theoretically substantiated. Finally, the effectiveness of the proposed scheme is demonstrated on a single integrator system, a nonlinear numerical system, and a unicycle kinematic system; besides, its advantages over the existing control methods are illustrated by comparisons.