Deep Reinforcement Learning and SQP-driven task offloading decisions in vehicular edge computing networks

Abstract

Vehicular Edge Computing offers low latency and reduced energy consumption for innovative applications through computation offloading in vehicular networks. However, making optimal offloading decisions and resource allocation remains challenging due to varying speeds, locations, channel quality constraints, and characteristics of both vehicles and tasks. To address these challenges, we propose a three-layered architecture and introduce a two-level algorithm named Sequential Quadratic Programming-based Dueling Double Deep Q Networks (SQ-DDTO) for optimal offloading actions and resource allocation. The joint computation offloading decision and resource allocation is a mixed integer nonlinear programming problem. To solve it, we first decouple the computation offloading decision sub-problem from resource allocation and address it using Dueling DDQN, which incorporates separate state values and action advantages. This decomposition allows for more granular control of computation tasks, leading to significantly better results. To enhance sample efficiency and learning in such complex networks, we employ Prioritized Experience Replay (PER). By prioritizing experiences based on their importance, PER enhances learning efficiency, allowing the agent to adapt quickly to changing conditions and optimize task offloading decisions in real time. Following this decomposition, we use Sequential Quadratic Programming (SQP) to solve for optimal resource allocation. SQP is chosen due to its effectiveness in handling non-convexity and complex constraints. Moreover, it has strong local convergence properties and utilizes gradient information which is crucial where rapid decision-making is necessary. Experimental results demonstrate the effectiveness of the proposed algorithm in terms of average delay, energy consumption, and task loss rate. For example. the proposed algorithm reduces the system cost by 25.1% compared to DQN and 16.67% compared to both DDQN and DDPG. Similarly. our method reduces the task loss rate by 37.06% compared to DQN, 34.78% compared to DDPG and 10.2% compared to DDQN.