CoSD: Balancing behavioral consistency and diversity in unsupervised skill discovery.

Abstract

In hierarchical reinforcement learning, unsupervised skill discovery holds promise for overcoming the challenge of sparse rewards commonly encountered in traditional reinforcement learning. Although previous unsupervised skill discovery methods excelled at maximizing intrinsic rewards, they often overly prioritized skill diversity. Unrestrained pursuit of diversity leads skills to concentrate attention on unexplored domains, overlooking the internal consistency of skills themselves, resulting in the state visit distribution of individual skills lacking concentration. To address this problem, the Constrained Skill Discovery (CoSD) algorithm is proposed to balance the diversity and behavioral consistency of skills. CoSD integrates both the forward and the reverse decomposition forms of mutual information and uses the maximum entropy policy to maximize the information-theoretic objective of skill learning while requiring that each skill maintain low state entropy internally, which enhances the behavioral consistency of the skills while pursuing the diversity of the skills and ensures that the learned skills have a high degree of stability. Experimental results demonstrated that, compared with other skill discovery methods based on mutual information, skills from CoSD exhibited a more concentrated state visit distribution, indicating higher behavioral consistency and stability. In some complex downstream tasks, the skills with higher behavioral consistency exhibit superior performance.