Solution Representations for Poisson’s Equation, Martingale Structure, and the Markov Chain Central Limit Theorem

The solution of Poisson’s equation plays a key role in constructing the martingale through which sums of Markov correlated random variables can be analyzed. In this paper, we study three different representations for the solution for countable state space irreducible Markov chains, two based on entry time expectations, and the other based on a potential kernel. Our consideration of null recurrent chains allows us to extend our theory to positive recurrent nonexplosive Markov jump processes. We also develop the martingale structure induced by these solutions to Poisson’s equation, under minimal conditions, and establish verifiable Lyapunov conditions to support our theory. Finally, we provide a central limit theorem for Markov dependent sums, under conditions weaker than have previously appeared in the literature.