Adaptive Multiagent Reinforcement Learning Solver for Tactical Conflict Resolution in Diverse Urban Airspace Configurations

Abstract

This article discusses the development of an adaptive deep reinforcement learning solver, designed for centralized on-ground tactical conflict resolution and applied within diverse in-cruise configurations of urban airspace. Our approach utilizes a multiagent reinforcement learning algorithm with a shared policy framework, employing a proximal policy optimization baseline. The solver aims to ensure tactical conflict resolution through centralized decision-making and distributed instructions via speed calibration and flight-level assignment. Each operation adheres to a designated flight plan and operational volume tolerances. Our methodology first enhances the solver's adaptability by interpreting a broad spectrum of in-cruise conflict configurations. This is achieved through a cautious observation and action space design, a normalization strategy, and a recurrent neural network that generalizes conflict resolution for any number of intruders. We also apply existing standards from airborne collision avoidance systems to evaluate the safety performance of our machine-learning-based solver, addressing the gap in machine learning model evaluation as a conventional system. We explore eight case studies of varying difficulty to investigate the safety impact of specific conflict configurations, from single to multiple intersections, considering shared routes or individual operation volumes, among other factors. Finally, we develop a ninth case study that maps all possible configurations under cruise conditions. We explore different trained models and assess their compatibility with other case studies, demonstrating the solver's adaptability in performing tactical conflict resolution tasks for operations in urban airspace.