Empirical risk minimization for dynamical systems and stationary processes

Abstract

We introduce and analyze a general framework for empirical risk minimization in which the observations and models of interest may be stationary systems or processes. Within the framework, which is presented in terms of dynamical systems, empirical risk minimization can be studied as a two-step procedure in which (i) the trajectory of an observed (but unknown) system is fit by a trajectory of a known reference system via minimization of cumulative per-state loss, and (ii) an invariant parameter estimate is obtained from the initial state of the best fit trajectory. We show that the weak limits of the empirical measures of best-matched trajectories are dynamically invariant couplings (joinings) of the observed and reference systems with minimal risk. Moreover, we establish that the family of risk-minimizing joinings is convex and compact and that it fully characterizes the asymptotic behavior of the estimated parameters, directly addressing identifiability. Our analysis of empirical risk minimization applies to well-studied problems such as maximum likelihood estimation and non-linear regression, as well as more complex problems in which the models of interest are stationary processes. To illustrate the latter, we undertake an extended analysis of system identification from quantized trajectories subject to noise, a problem at the intersection of dynamics and statistics.