An advantage duPLEX dueling multi-agent Q-learning algorithm for multi-UAV cooperative target search in unknown environments

Abstract

Multiple unmanned aerial vehicles (UAVs) cooperative target search has been extensively applied in post-disaster relief and surveillance tasks. However, achieving efficient cooperative target search in unknown environments without prior information is extremely challenging. In the study, a novel multi-agent deep reinforcement learning (MADRL)-based approach is proposed to enable UAVs to execute target search in the three-dimensional (3D) unknown environments. Specifically, the target search problem is formulated as a decentralized partially observable Markov decision processes (Dec-POMDP), where each UAV maintains its own target existence probability map and merges with those of other UAVs within communication range to enhance UAVs’ perception of task environment. Then, an improved duPLEX dueling multi-agent Q-learning (QPLEX) algorithm called Advantage QPLEX is proposed to make the optimal decision for multiple UAVs target search. The Advantage QPLEX can guide UAVs to focus on the advantage steps during the search to improve search efficiency, and direct UAVs to select the advantage action in each step for a greater return. In addition, a novel reward function is well-designed for cooperative target search problems to drive UAVs to explore and utilize the environmental information efficiently. Extensive simulations conducted on the Airsim validate that the Advantage QPLEX outperforms the existing algorithms in terms of the coverage rate and search rate.