Convergence of policy gradient for stochastic linear quadratic optimal control problems in infinite horizon

Recently, there has been an increasing interest in studying theoretical properties of gradient-based methods in the context of control and reinforcement learning. This article studies the policy gradient (PG) method for infinite horizon continuous-time stochastic linear quadratic (SLQ) optimal control problems, where the drift and diffusion terms in dynamics depend on both the state and control. Within the LQ framework, the optimal controls can usually be expressed by a linear state feedback form. Thus we formulate the SLQ problem as a policy optimization (PO) problem, where the policy is linearly parameterized. A main challenge for the PO formulation is that the stability constraints are nonconvex in the policy space. We overcome this difficulty by leveraging the gradient domination condition and L-smoothness property, thereby establishing global exponential/linear convergence of the gradient flow/descent algorithm. Finally, a numerical example is given to illustrate the convergence of gradient descent method.