Mild evaluation policy via dataset constraint for offline reinforcement learning

Abstract

Offline reinforcement learning (RL) agents seek optimal policies from fixed datasets. Policy constraints are typically employed to adhere to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions. Conventional approaches apply identical constraints for both value learning and test time inference. However, the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a mild evaluation policy via dataset constraint (MEDC) for test time inference with a more constrained target policy for value estimation. MEDC introduces a dual-policy constraint, comprising a target policy and an evaluation policy. The evaluation policy regularize the policy towards the nearest state–action pair, with behavior cloning performed on the target policy. The distributional shift is effectively addressed through the combination of dataset constraint and behavior cloning. The TD3 is employed to direct the policy in selecting actions that maximize the return. Moreover, MEDC achieves state-of-the-art performance compared with existing methods on the D4RL datasets.