Bivariate Markov Processes and Their Estimation

Abstract

A bivariate Markov process comprises a pair of random processes which are jointly Markov. One of the two processes in that pair is observable while the other plays the role of an underlying process. We are interested in three classes of bivariate Markov processes. In the first and major class of interest, the underlying and observable processes are continuous-time with finite alphabet; in the second class, they are discrete-time with finite alphabet; and in the third class, the underlying process is continuous-time with uncountably infinite alphabet, and the observable process is continuous-time with countably or uncountably infinite alphabet. We refer to processes in the first two classes as bivariate Markov chains. Important examples of continuoustime bivariate Markov chains include the Markov modulated Poisson process, and the batch Markovian arrival process. A hidden Markov model with finite alphabet is an example of a discrete-time bivariate Markov chain. In the third class we have diffusion processes observed in Brownian motion, and diffusion processes modulating the rate of a Poisson process. Bivariate Markov processes play central roles in the theory and applications of estimation, control, queuing, biomedical engineering, and reliability. We review properties of bivariate Markov processes, recursive estimation of their statistics, and recursive and iterative parameter estimation.