Resources allocation for underwater acoustic soft frequency reuse network based on multi-agent deep reinforcement learning

Abstract

The bandwidth and power resources in underwater acoustic sensor networks (UASNs) are severely limited. By adopting adaptive resource allocation technique, the network capacity and energy efficiency of UASNs can be improved. In this paper, we model the underwater acoustic (UWA) soft frequency reuse (SFR) network as a multi-agent system, and propose a multi-agent deep Q network based resource allocation (MADQN-RA) method. The system state is designed as outdated feedback channel state information (CSI) sequences, considering the time-varying and long propagation delay features of UWA channel. By establishing an effective joint reward expression, the intelligent agents can mapping the relationship of state–action and reward in time-varying UWA channel and make corresponding resource allocation decisions. Furthermore, to improve the learning efficiency, a dynamic state length method is proposed with the specific design of multi-stage experience buffer. The pre-training method is also combined for further improvement of system efficiency. Simulation results show that the system performance of the proposed methods is better than other learning-based methods and channel prediction-based methods, and is closer to the theoretical optimal value.