Energy-Efficient Multi-Agent Reinforcement Learning for UAV Trajectory Optimization in Cell-Free Massive MIMO Networks

Abstract

To enhance global data transmission, uncrewed aerial vehicle (UAV)-aided space-air-ground integrated networks (SAGIN) represent a pivotal direction for future advancements. In this paper, we focus on the trajectory optimization problem with the goal of maximizing the energy efficiency (EE), thereby balancing the system capacity with energy expenditure. To this end, we first introduce a cell-free SAGIN network where UAVs function as flying access points to serve ground user equipment (GUE). Given that the transmission power of satellite direct-to-cell devices typically exceeds that of GUEs, we investigate the interference effect and derive exact closed-form expressions for the uplink spectral efficiency. In order to improve the service access efficiency, a GUE grouping scheme based on density distribution is proposed. Then, an effective EE analysis model is established considering the power consumption of fixed-wing UAVs. To solve the UAV trajectory optimization problem, two algorithms over two timescales are proposed: a successive convex approximation strategy and a multi-agent reinforcement learning (MARL)-based algorithm. In particular, to reduce the algorithmic complexity, we employ a shared Critic network in the proposed MARL algorithm to reduce the training parameters. Importantly, our approach comprehensively optimizes the UAV trajectory, acceleration, and velocity parameters. The results show that the proposed GUE grouping algorithm and the MARL-based optimization algorithm demonstrate adaptability in dynamic time-varying environments.