Swap Softmax Twin Delayed Deep Deterministic Policy Gradient

Abstract

Reinforcement learning algorithms have attained noteworthy accomplishments in the field of continuous control. One of the classic algorithms in continuous control, the DDPG algorithm, is widely used and has been shown to be susceptible to overestimation. Following this, the TD3 algorithm was introduced, which integrated the notion of double DQN. TD3 takes into account the minimum value between a pair of critics to restrict overestimation. Nevertheless, TD3 may lead to an underestimation bias. To mitigate the impact of errors, we present a novel approach by integrating Swap Softmax with TD3, which can counterbalance the extreme values. We assess the efficacy of our proposed technique on continuous control tasks that are simulated by MuJoCo and provided by OpenAI Gym. Our experimental findings demonstrate a significant enhancement in the performance and robustness.