{"title":"Likelihood Ratio Gradient Estimation for Steady-State Parameters","authors":"P. Glynn, Mariana Olvera-Cravioto","doi":"10.1287/STSY.2018.0023","DOIUrl":null,"url":null,"abstract":"We consider a discrete-time Markov chain $\\boldsymbol{\\Phi}$ on a general state-space ${\\sf X}$, whose transition probabilities are parameterized by a real-valued vector $\\boldsymbol{\\theta}$. Under the assumption that $\\boldsymbol{\\Phi}$ is geometrically ergodic with corresponding stationary distribution $\\pi(\\boldsymbol{\\theta})$, we are interested in estimating the gradient $\\nabla \\alpha(\\boldsymbol{\\theta})$ of the steady-state expectation $$\\alpha(\\boldsymbol{\\theta}) = \\pi( \\boldsymbol{\\theta}) f.$$ \nTo this end, we first give sufficient conditions for the differentiability of $\\alpha(\\boldsymbol{\\theta})$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.","PeriodicalId":36337,"journal":{"name":"Stochastic Systems","volume":" ","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2017-07-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1287/STSY.2018.0023","citationCount":"8","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Stochastic Systems","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1287/STSY.2018.0023","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"Mathematics","Score":null,"Total":0}
引用次数: 8
Abstract
We consider a discrete-time Markov chain $\boldsymbol{\Phi}$ on a general state-space ${\sf X}$, whose transition probabilities are parameterized by a real-valued vector $\boldsymbol{\theta}$. Under the assumption that $\boldsymbol{\Phi}$ is geometrically ergodic with corresponding stationary distribution $\pi(\boldsymbol{\theta})$, we are interested in estimating the gradient $\nabla \alpha(\boldsymbol{\theta})$ of the steady-state expectation $$\alpha(\boldsymbol{\theta}) = \pi( \boldsymbol{\theta}) f.$$
To this end, we first give sufficient conditions for the differentiability of $\alpha(\boldsymbol{\theta})$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.