Reinforcement learning-based motion planning in partially observable environments under ethical constraints

Abstract

Designing autonomous agents that follow moral norms presents a significant challenge in addressing AI decision-making under ethical constraints, especially when involving motion planning for complex tasks in partially observable environments. This paper proposes a model-free reinforcement learning approach to address these challenges. We formulate the motion planning problem as a Probabilistic-Labeled Partially Observable Markov Decision Process (PL-POMDP) model and express complex tasks using Linear Temporal Logic (LTL). To handle ethical norms, we categorize them into ‘hard’ and ‘soft’ ethical constraints. LTL is again employed to formulate ‘hard’ constraints, while a reward redesign method is applied to enforce ‘soft’ ethical constraints. Our approach also involves generating a product of PL-POMDP and an LTL-induced automaton. This transformation allows us to find an optimal policy on the product, ensuring both task completion and ethics satisfaction through model checking. To synthesize desired policies, we utilize a state-of-the-art Recurrent Neural Network (RNN)-based deep Q learning method, in which Q networks take into account observation history and task recognition as input features. We demonstrate the effectiveness and flexibility of the proposed approach through two simulation examples, which showcase its potential applicability to various scenarios and challenges in ethically guided AI decision-making.