Robust estimation of a regression function in exponential families

Abstract

We observe $n$ pairs of independent (but not necessarily i.i.d.) random variables $X_1=(W_1,Y_1),\dots ,X_n=(W_n,Y_n)$ and tackle the problem of estimating the conditional distributions $Q_i^{\star }\left(w_i\right)$ of $Y_i$ given $W_i=w_i$ for all $i\in \{1,\dots ,n\}$. Even though these might not be true, we base our estimator on the assumptions that the data are i.i.d. and the conditional distributions of $Y_i$ given $W_i=w_i$ belong to a one parameter exponential family $\overline{Q}$ with parameter space given by an interval $I$. More precisely, we pretend that these conditional distributions take the form $Q_{\theta \left(w_i\right)}\in \overline{Q}$ for some $\theta$ that belongs to a VC-class $\overline{\Theta }$ of functions with values in $I$. For each $i\in \{1,\dots ,n\}$, we estimate $Q_i^{\star }\left(w_i\right)$ by a distribution of the same form, i.e. $Q_{\hat{\theta }\left(w_i\right)}\in \overline{Q}$, where $\hat{\theta }=\hat{\theta }(X_1,\dots ,X_n)$ is a well-chosen estimator with values in $\overline{\Theta }$. We establish non-asymptotic exponential inequalities for the upper deviations of a Hellinger-type distance between the true conditional distributions of the data and the estimated one based on the exponential family $\overline{Q}$ and the class of functions $\overline{\Theta }$ we chose. We show that our estimation strategy is robust to model misspecification, contamination and the presence of outliers. Besides, when the data are truly i.i.d., the exponential family $\overline{Q}$ is suitably parametrized and the conditional distributions $Q_i^{\star }\left(w_i\right)$ of the form $Q_{{\theta }^{\star }\left(w_i\right)}\in \overline{Q}$ for some unknown Hölderian function ${\theta }^{\star }$ with values in $I$, we prove that the estimator $\hat{\theta }$ of ${\theta }^{\star }$ is minimax (up to a logarithmic factor). Finally, we provide an algorithm for calculating $\hat{\theta }$ when $\overline{\Theta }$ is a VC-class of functions of low or moderate dimension and we carry out a simulation study to compare its performance to that of the MLE and median-based estimators. The proof of our main result relies on an upper bound, with explicit numerical constants, on the expectation of the supremum of an empirical process over a VC-subgraph class. This bound can be of independent interest.