Active Robust Adversarial Reinforcement Learning Under Temporally Coupled Perturbations

Abstract

Robust reinforcement learning (RL) aims to improve the generalization of agents under model mismatch. As a major branch of robust RL, adversarial approaches formulate the problem as a zero-sum game in which adversaries seek to apply worst case perturbations to the dynamics. However, the potential constraints of adversarial perturbations are seldom addressed in existing approaches. In this article, we consider temporally coupled settings, where adversarial perturbations change continuously at a bounded rate. This kind of constraint can commonly arise in a variety of real-world situations (e.g., changes in wind speed and ocean currents). We propose a novel robust RL approach, named active robust adversarial RL (ARA-RL), that tackles this problem in an adversarial architecture. First, we introduce a type of RL adversary that generates temporally coupled perturbations on agent actions. Then, we embed a diagnostic module in the RL agent, enabling it to actively detect temporally coupled perturbations in unseen environments. Through adversarial training, the agent seeks to maximize its worst case performance and thus achieve robustness under perturbations. Finally, extensive experiments demonstrate that our proposed approach provides significant robustness against temporally coupled perturbations and outperforms other baselines on several continuous control tasks.