Optimal control of time-inhomogeneous Markov chains with application to dam management

Abstract

We consider a time-inhomogeneous Markov chain with a compound Poisson process as an input as an important approximation to get the solution of real problems. Outflows are comprised of both controlled and uncontrolled counting processes. We demonstrate the utility of this model in the specific problem of demand control and flood prevention in a nearly full dam, where the dam is modeled as a continuous-time controllable Markov chain under control resource constraints. This work significantly extends previous results because the inflow of rain is potentially sudden and of large volume, so a compound Poisson process is preferred to simple counting processes. On the other hand the safe release of water is constrained and so is modelled as a simple counting process. A proof of the infinitesimal generator of the Markov chain with these characteristics is given and a numerical example demonstrates the effectiveness of the controls.