Many-Versus-Many AUV Attack-Defense Game in 3-D Scenarios Using Hierarchical Multiagent Reinforcement Learning

Abstract

This article proposes a deep reinforcement learning (DRL)-based method for many-versus-many attack-defense games involving autonomous underwater vehicles (AUVs) in 3-D space, focusing on training a defense team to counter attackers. The attackers benefit from speed and unpredictability, while defenders leverage numerical superiority. The scenario includes irregular terrain, and AUVs are limited by low-frequency communication. First, a constrained Apollonius model considering AUV 3-D motion characteristics is developed to evaluate the repulsive effect of defenders on attackers. Second, a hybrid 3-D AUV maneuvering framework integrating end-to-velocity and hierarchical approaches is proposed to reduce the complexity of decision-making strategy learning, enabling AUVs to counter multiattacker threats and learn repulsion strategies across subteams. Third, a scalable learning architecture is designed to adapt to different game scales, with an improved update method to enhance advantage and credit estimation efficiency while ensuring convergence. The combination of population expansion-curriculum training and asynchronous parallel training strengthens the generalization of strategies across various environments. Finally, through comparative analysis with mainstream multiagent deep reinforcement learning-based methods, as well as ablation studies on the framework and rewards, our scheme demonstrates superior learning efficiency and generalization ability. Adversarial experiments across different game scales, along with specialized performance tests, indicate that the defense group exhibits strong robustness and adaptive characteristics.