Digital Twin-Enabled Deep Reinforcement Learning for Safety-Guaranteed Flocking Motion of UAV Swarm

Abstract

Multi-agent deep reinforcement learning (MADRL) has become a typical paradigm for the flocking motion of UAV swarm in dynamic, stochastic environments. However, sim-to-real problems, such as reality gap, training efficiency, and safety issues, restrict the application of MADRL in flocking motion scenarios. To address these problems, we first propose a digital twin (DT)-enabled training framework. With the assistance of high-fidelity digital twin simulation, effective policies can be efficiently trained. Based on the multi-agent proximal policy optimization (MAPPO) algorithm, we then design the learning approach for flocking motion with matching observation space, action space, and reward function. Afterward, we employ a distributed flocking center estimation algorithm based on position consensus. The estimated center is used as a policy input to improve the aggregation behavior. Moreover, we introduce a repulsion scheme, which applies an additional repulsion force to the action to prevent UAVs from colliding with neighbors and obstacles. Simulation results show that our method performs well in maintaining flocking formation and avoiding collisions, and has better decision-making ability in near-realistic environments.