Joint optimization for service-caching, computation-offloading, and UAVs flight trajectories over rechargeable UAV-aided MEC using hierarchical multi-agent deep reinforcement learning

Due to the high mobility, high chance of line-of-sight (LoS) transmission, and flexible deployment, unmanned aerial vehicles (UAVs) have been used as mobile edge computing (MEC) servers to provide ubiquitous computation services to mobile users (MUs). However, the limited energy storage, caching capacity, and computation resources of UAVs bring new challenges for UAV-aided MEC, e.g., how to recharge UAVs and how to jointly optimize service-caching, computation-offloading, and UAVs flight trajectories. To overcome the above-mentioned difficulties, in this paper we study the joint optimization for service-caching, computation-offloading, and UAVs flight trajectories for UAV-aided MEC, where multiple rechargeable UAVs cooperatively provide MEC services to a number of MUs. First, we formulate an energy minimization problem to minimize all MUs' energy consumptions by taking into account the mobility of MUs and the energy replenishment of UAVs. Then, using the hierarchical multi-agent deep reinforcement learning (HMDRL), we develop a two-timescale based joint service-caching, computation-offloading, and UAVs flight trajectories scheme, called HMDRL-Based SCOFT. Using HMDRL-Based SCOFT, we derive UAVs' service-caching policies in each time frame, and then derive UAVs flight trajectories and MUs' computation-offloading in each time slot. Finally, we validate and evaluate the performances of our proposed HMDRL-Based SCOFT scheme through extensive simulations, which show that our developed scheme outperforms the other baseline schemes to converge faster and greatly reduce MUs' energy consumptions.