Robust Q-learning for Fast And Optimal Flying Base Station Placement Aided By Digital Twin For Emergency Use

Abstract

This paper studies a use case for quickly connecting users with an aerial bass station (BS) in emergency by leveraging digital twin (DT) and robust reinforcement learning (RL). Scattered communities of users are assigned a limited number of channels, and the flying BS is autonomously and optimally placed according to predefined criteria. Q-learning, a common type of RL, is employed as a solution to optimization of BS placement. Two optimization objectives are considered to maximize the per-user data rate in the worst condition and minimize the total BS transmitted power, respectively. To overcome resource limitations of the aerial BS, the RL training with many iterations is done in the DT virtual space connected to the physical space via an aerial BS. In particular, a practically sound max-min technique is proposed to handle the measurement and prediction uncertainties. It is shown that, even if the models used in DT are imperfect and measurements are inaccurate, nearly-optimal results can be obtained in DT. Compared to RL training 100% in the physical space, a huge number of BS moves can be avoided and a significant amount of time and energy can be saved. The assessment results suggest that model and measurement errors, especially when applying DT, should be seriously considered, and the robust optimization technique has potential to handle the uncertainties.