HiPPO: Enhancing proximal policy optimization with highlight replay

Abstract

Sample efficiency remains a paramount challenge in policy gradient methods within reinforcement learning. The success of experience replay demonstrates the importance of leveraging historical experiences, often through off-policy methods to enhance approximate policy learning algorithms that aim to maximize current interaction sample reuse, aligning approximate policies with target objectives. However, the inaccurate approximation can negatively affect actual optimization, leading to poorer current experiences than past ones. We propose Highlight Replay Enhanced Proximal Policy Optimization (HiPPO) to address the challenge. Specifically, HiPPO optimizes by highlighting policies and introducing a penalty reward function for constrained optimization, which alleviates the constraints of policy similarity and boosts adaptability to historical experiences. Empirical studies show HiPPO outperforming state-of-the-art algorithms in MuJoCo continuous tasks in performance and learning speed. An in-depth analysis of the experimental results validates the effectiveness of employing highlight replay and penalty reward functions in our proposed method.