Research on Delay DRL in Energy-Constrained CR-NOMA Networks based on Multi-Threads Markov Reward Process

Applying deep reinforcement learning in wireless networks has been a hot topic in the field of non-orthogonal multiple access. Most present works focus on the design of algorithms and ignore one of the practical problems when deploying them in actual networks: the computing delay, which may lead to performance deterioration. In this paper, we focus on DDPG applied in energy-constrained CR-NOMA networks with delays and propose a multi-threads scheme to assist the main agent to select action in time. We first discuss the workflow of the proposed scheme, and then restore the impaired Markovianity due to the introduction of subthreads by enhancing the state space in MRP. Test results show that the proposed scheme can significantly improve the performance of DDPG in CR-NOMA networks with delays.