Approximate Constrained Discounted Dynamic Programming With Uniform Feasibility and Optimality

Abstract

An important question about finite constrained Markov decision process (CMDP) problem is if there exists a condition under which a uniformly optimal and uniformly feasible policy exists in the set of deterministic, history-independent, and stationary policies that achieves the optimal value at all initial states and if the CMDP problem with the condition can be solved by dynamic programming (DP). This is because the crux of the unconstrained MDP theory developed by Bellman lies in the answer to the same existence question of such an optimal policy to MDP. Even if the topic of CMDP has been studied over the years, there has not been any relevant responsive work since the open question was raised about three decades ago in the literature. We establish (as some answer to this question) that any finite CMDP problem $ \mathsf{M}^{c}$ “contains” inherently a DP-structure in its “subordinate” CMDP problem $\hat{ \mathsf{M} }^{c}$ induced from the parameters of $ \mathsf{M} ^{c}$ and $\hat{\mathsf{M} }^{c}$ is DP-solvable. We drive a policy-iteration-type algorithm for solving $\hat{\mathsf{M} }^{c}$ providing an approximate solution to $ \mathsf{M}^{c}$ or $ \mathsf{M} ^{c}$ with a fixed initial state.