Throughput Maximization in Delay-Critical and Energy-Aware SW-UAV-WNs Using Q-Learning

Unmanned aerial vehicles (UAVs) are getting significant attention from both researchers and the industry due to their wide range of applications. Remote sensing is one such application, in which UAVs are deployed to sense remote areas and transmit the data to a ground station for processing. However, due to the mobility and limited transmission range of UAVs, data transfer requires multiple hops. Nevertheless, the higher the number of hops, the larger the network latency. Thus, there is a need to reduce the number of hops and improve the connectivity. This can be achieved by creating small-world networks (SWNs) that perform better than traditional networks in terms of network evaluation metrics. The SWNs are created by adding shortcuts to the traditional network. In the literature, many theoretical works have been proposed for the creation of SWNs. However, these works add shortcuts randomly into the existing conventional network and fail to account for the costs incurred with the added shortcuts. As a result, these works are ineffective in improving the overall performance of the network. Thus, this work presents a novel reinforcement learning technique that uses a Q-learning algorithm to optimize throughput in delay-critical and energy-aware small-world UAV-assisted wireless networks (SW-UAV-WNs). The proposed algorithm populates the Q-matrix with all possible shortcuts and updates the Q-values based on the reward/penalty. It then adds shortcuts based on descending Q-values until the SW-UAV-WN is established. Through numerical results, we demonstrate that the proposed framework surpasses the conventional SWC approach, canonical particle swarm data delivery method, Low Energy Adaptive Clustering Hierarchy (LEACH), modified LEACH, and conventional shortest path routing method in terms of network latency, lifetime, packet delivery ratio, and throughput. Furthermore, we discuss the effect of different UAV velocities and different heights of the layers in which the UAVs hover on the performance of the proposed approach.