Continuous-Time Markov Chains

A Markov chain in discrete time, {X n : n ≥ 0}, remains in any state for exactly one unit of time before making a transition (change of state). We proceed now to relax this restriction by allowing a chain to spend a continuous amount of time in any state, but in such a way as to retain the Markov property. As motivation, suppose we consider the rat in the open maze. Clearly it is more realistic to be able to keep track of where the rat is at any continuous-time t ≥ 0 as oppposed to only where the rat is after n " steps ". Assume throughout that our state space is S = Z = {· · · , −2, −1, 0, 1, 2, · · · } (or some subset thereof). Suppose now that whenever a chain enters state i ∈ S, independent of the past, the length of time spent in state i is a continuous, strictly positive (and proper) random variable H i called the holding time in state i. When the holding time ends, the process then makes a transition into state j according to transition probability P ij , independent of the past, and so on. 1 Letting X(t) denote the state at time t, we end up with a continuous-time stochastic process {X(t) : t ≥ 0} with state space S. Our objective is to place conditions on the holding times to ensure that the continuous-time process satisfies the Markov property: The future, {X(s + t) : t ≥ 0}, given the present state, X(s), is independent of the past, {X(u) : 0 ≤ u < s}. Such a process will be called a continuous-time Markvov chain (CTMC), and as we will conclude shortly, the holding times will have to be exponentially distributed. The formal definition is given by Definition 1.1 A stochastic process {X(t) : t ≥ 0} with discrete state space S is called a continuous-time Markvov chain (CTMC) if for all t ≥ 0, s ≥ 0, i ∈ S, j ∈ S, P (X(s + t) = j|X(s) = i, {X(u) : 0 ≤ u < s}) = P (X(s + t) = j|X(s) = i) = P ij (t). P ij (t) is the probability that the chain will be in state j, t time units from now, given it is in state i now. For …