Knowledge guided deep deterministic policy gradient

Abstract

Deep deterministic policy gradient (DDPG) exhibits excellent handling capabilities for complex regulation and control problems with continuous state and action spaces. However, its trial-and-error interaction and learning from scratch require extensive exploration by the agent, leading to low learning efficiency and even non-convergence in sparse reward environments. To fully utilize knowledge during the learning process to improve efficiency and performance, this paper draws inspiration from human learning methods and proposes a semantic knowledge-guided DDPG (KGDDPG) approach. In terms of knowledge representation, considering the fuzziness and precision of semantic knowledge, a knowledge system based on a rule framework combining precise propositions and fuzzy propositions is constructed. In terms of knowledge integration, to reduce the randomness of exploration, a knowledge-guided action strategy based on stacked generalization is proposed. Furthermore, a supervised-then-reinforced learning method is employed: the ”supervised” phase quickly incorporates prior knowledge to accelerate learning, while the ”reinforced” phase refines the policy network to overcome the limitations of relying solely on prior knowledge. Finally, experiments were conducted using a mapless navigation task for mobile robots to verify the effectiveness and practical feasibility of the method.