Pub Date : 2024-11-29DOI: 10.1109/OJCOMS.2024.3509777
Fahri Wisnu Murti;Samad Ali;Matti Latva-Aho
Multi-access Edge Computing (MEC) can be implemented together with Open Radio Access Network (O-RAN) to offer low-cost deployment and bring services closer to end-users. In this paper, the joint orchestration of O-RAN and MEC using a Bayesian deep reinforcement learning (RL) framework is proposed. The framework jointly controls the O-RAN functional splits, O-RAN/MEC computing resource allocation, hosting locations, and data flow routing across geo-distributed platforms. The goal is to minimize the long-term total network operation cost and maximize MEC performance criterion while adapting to varying demands and resource availability. This orchestration problem is formulated as a Markov decision process (MDP). However, finding the exact model of the underlying O-RAN/MEC system is impractical since the system shares the same resources, serves heterogeneous demands, and its parameters have non-trivial relationships. Moreover, the formulated MDP results in a large state space with multidimensional discrete actions. To address these challenges, a model-free RL agent based on a combination of Double Deep Q-network (DDQN) with action branching is proposed. Furthermore, an efficient exploration-exploitation strategy under a Bayesian learning framework is leveraged to improve learning performance and expedite convergence. Trace-driven simulations are performed using an O-RAN-compliant model. The results show that our approach is data-efficient (i.e., converges significantly faster), increases the reward by 32% compared to its non-Bayesian version, and outperforms Deep Deterministic Policy Gradient by up to 41%.
{"title":"A Bayesian Framework of Deep Reinforcement Learning for Joint O-RAN/MEC Orchestration","authors":"Fahri Wisnu Murti;Samad Ali;Matti Latva-Aho","doi":"10.1109/OJCOMS.2024.3509777","DOIUrl":"https://doi.org/10.1109/OJCOMS.2024.3509777","url":null,"abstract":"Multi-access Edge Computing (MEC) can be implemented together with Open Radio Access Network (O-RAN) to offer low-cost deployment and bring services closer to end-users. In this paper, the joint orchestration of O-RAN and MEC using a Bayesian deep reinforcement learning (RL) framework is proposed. The framework jointly controls the O-RAN functional splits, O-RAN/MEC computing resource allocation, hosting locations, and data flow routing across geo-distributed platforms. The goal is to minimize the long-term total network operation cost and maximize MEC performance criterion while adapting to varying demands and resource availability. This orchestration problem is formulated as a Markov decision process (MDP). However, finding the exact model of the underlying O-RAN/MEC system is impractical since the system shares the same resources, serves heterogeneous demands, and its parameters have non-trivial relationships. Moreover, the formulated MDP results in a large state space with multidimensional discrete actions. To address these challenges, a model-free RL agent based on a combination of Double Deep Q-network (DDQN) with action branching is proposed. Furthermore, an efficient exploration-exploitation strategy under a Bayesian learning framework is leveraged to improve learning performance and expedite convergence. Trace-driven simulations are performed using an O-RAN-compliant model. The results show that our approach is data-efficient (i.e., converges significantly faster), increases the reward by 32% compared to its non-Bayesian version, and outperforms Deep Deterministic Policy Gradient by up to 41%.","PeriodicalId":33803,"journal":{"name":"IEEE Open Journal of the Communications Society","volume":"5 ","pages":"7685-7700"},"PeriodicalIF":6.3,"publicationDate":"2024-11-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10771978","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142825903","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Recent advancements in low Earth orbit (LEO) satellite technology have facilitated a substantial increase in the number of Earth observation (EO) satellites launched. However, transmitting voluminous imagery generated by these EO satellites to the ground still faces the challenges of limited satellite resources and dynamic satellite networks. To address this problem, we propose SEC-DT, a S