Deep multi-agent RL for anti-jamming and inter-cell interference mitigation in NOMA networks

Inter-cell interference and smart jammer attacks significantly impair the performance of non-orthogonal multiple access (NOMA) networks. This issue is particularly critical when considering strategic interactions with malicious actors. To address this challenge, the power allocation problem is framed in a two-cell NOMA network as a sequential game. In this game, each base station acts as a leader, choosing a power allocation strategy, while the smart jammer acts as a follower, reacting optimally to the base stations' choices. To address this multi-agent scenario, four multi-agent reinforcement learning algorithms are proposed: Q-learning based unselfish (QLU), deep QLU, hot booting deep QLU, and decreased state deep QLU. A game-theoretic analysis that demonstrates the algorithms' convergence to the optimal network-wide strategy with high probability is provided. Simulation results further confirm the superiority of our proposed algorithms compared to the Q-learning-based selfish NOMA power allocation method.