Set-based value operators for non-stationary and uncertain Markov decision processes

Abstract

This paper analyzes finite-state Markov Decision Processes (MDPs) with nonstationary and uncertain parameters via set-based fixed point theory. Given compact parameter ambiguity sets, we demonstrate that a family of contraction operators, including the Bellman operator and the policy evaluation operator, can be extended to set-based contraction operators with a unique fixed point—a compact value function set. For non-stationary MDPs, we show that while the value function trajectory diverges, its Hausdorff distance from this fixed point converges to zero. In parameter uncertain MDPs, the fixed point’s extremum value functions are equivalent to the min–max value function in robust dynamic programming under the rectangularity condition. Furthermore, we show that the rectangularity condition is a sufficient condition for the fixed point to contain its own extremum value functions. Finally, we derive novel guarantees for probabilistic path planning in capricious wind fields and stratospheric station-keeping.