GSADDQN: Combining GraphSAGE and reinforcement learning for routing optimization in software-defined optical transport network

Abstract

As a result of rapid development of network communication technology, optical transport network (OTN) traffic has also experienced rapid growth in terms of information volume scale, traffic complexity, and spatiotemporal distribution dynamics. Because the OTN traffic demand has complex spatiotemporal fluctuations, the traditional deep reinforcement learning (DRL) algorithm has been applied to the routing optimization of software-defined OTNs. However, the traditional DRL algorithm has problems such as slow convergence, weak generalization ability, and load imbalance when performing routine tasks. To address these issues, we propose a graph sampling and aggregation (GraphSAGE)-based dueling deep Q-network (GSADDQN) algorithm for software-defined OTN routing optimization. First, we design a DRL-based routing decision model to find the best routing strategy for each optical network’s source–destination traffic demand. Second, considering the sparse connection characteristics of optical network nodes, we use sampling neighbors and a deep aggregation mechanism as the neural network model of the Dueling Deep Q-Network (Dueling DQN) algorithm so that the reinforcement learning agent can consciously aggregate important network information and improve the model’s convergence performance and generalization ability. Finally, we design simulation routing experiments based on Gym and evaluate the algorithm’s load balancing and generalization capabilities for different network topologies. The experimental results show that the GSADDQN algorithm has good convergence performance and load balancing ability in routing optimization of optical transmission networks and can generalize new network structures, maintaining good decision-making ability even during network node failures.