Unified finite-time error analysis of soft Q-learning

Abstract

Soft Q-learning is one of the most commonly used reinforcement learning algorithms for various purposes, e.g., dealing with entropy-regularized Markov decision problems, reducing the overestimation bias, and improving explorations. Its effectiveness in practice has led to its widespread use; however, there has not been much theoretical study on soft Q-learning. This paper attempts to provide an integrated finite-time analytical approach for soft Q-learning from a control-theoretic perspective. We examine three different kinds of soft Q-learning algorithms that use the log-sum-exp operator, the Boltzmann operator, and the mellowmax operator, respectively. Utilizing dynamical switching system models, we obtain the finite-time error bounds of three soft Q-learning variants. We believe that our analysis can assist in a better understanding of soft Q-learning through links with switching system models.