Probabilistic Multi-knowledge Transfer in Reinforcement Learning

Abstract

Transfer in Reinforcement Learning (RL) aims to remedy the problem of learning complex RL tasks from scratch, which is impractical in most of the cases due to the huge sample requirements. To overcome this problem, transferring the knowledge acquired from a set of source tasks to a new target task is a core idea. This knowledge can be the policy, the model (state transition and/or reward function), or the value function learned in the source tasks. However, algorithms in transfer learning focus on transferring a single type of knowledge at a time, although intuitively it might be interesting to reuse several types of this knowledge. For this reason, in this paper we propose a multi-knowledge transfer RL algorithm which we call Probabilistic Transfer of Policies and Models (PTPM). PTPM, unlike single-knowledge transfer approaches, combines the transfer of two types of knowledge: policies and models. We show through different experiments on two well-known domains (Grid World and Mountain Car) how this novel multi-knowledge transfer algorithm improves the results of the two methods in which it is inspired separately. As an additional result, we show that sequential learning of multiple tasks is generally better than learning from a library of previously learned tasks from scratch.