UAV Assisted Cellular Networks With Renewable Energy Charging Infrastructure: A Reinforcement Learning Approach

Abstract

Deploying unmanned aerial vehicle (UAV) mounted base stations with a renewable energy charging infrastructure in a temporary event (e.g., sporadic hotspots for light reconnaissance mission or disaster-struck areas where regular power-grid is unavailable) provides a responsive and cost-effective solution for cellular networks. Nevertheless, the energy constraint incurred by renewable energy (e.g., solar panel) imposes new challenges on the recharging coordination. The amount of available energy at a charging station (CS) at any given time is variable depending on: the time of day, the location, sunlight availability, size and quality factor of the solar panels used, etc. Uncoordinated UAVs make redundant recharging attempts and result in severe quality of service (QoS) degradation. The system stability and lifetime depend on the coordination between the UAVs and available CSs. In this paper, we develop a reinforcement learning time-step based algorithm for the UAV recharging scheduling and coordination using a Q-Learning approach. The agent is considered a central controller of the UAVs in the system, which uses the $\epsilon$-greedy based action selection. The goal of the algorithm is to maximize the average achieved throughput, reduce the number of recharging occurrences, and increase the life-span of the network. Extensive simulations based on experimentally validated UAV and charging energy models reveal that our approach exceeds the benchmark strategies by 381% in system duration, 47% reduction in the number of recharging occurrences, and achieved 66% of the performance in average throughput compared to a power-grid based infrastructure where there are no energy limitations on the CSs.