Metric Interval Temporal Logic based Reinforcement Learning with Runtime Monitoring and Self-Correction

Abstract

In this paper we present a modular Q-learning framework to deal with the robot task planning, runtime monitoring and self-correction problem. The task is specified using metric interval temporal logic (MITL) with finite time constraints. We first construct a runtime monitor automaton using three-valued LTL (LTL3), and a sub-task MITL monitor is constructed by decomposing and augmenting the monitor automaton. During the learning phase, a modular Q-learning approach is proposed such that each module could learn different sub-tasks. During runtime, the sub-task MITL monitors could monitor the execution and guide the agent for possible self-correction if an error occurs. Our experiments show that under our framework, the robot is able to learn a feasible execution sequence that satisfies the given MITL specifications under finite time constraints. When the runtime environment becomes different than the learning environment and the original action will violate the specifications, the robotic agent is able to self-correct and accomplish the task if it is still possible.