Universality in Block Dependent Linear Models With Applications to Nonlinear Regression

Abstract

Over the past decade, characterizing the precise asymptotic risk of regularized estimators in high-dimensional regression has emerged as a prominent research area. This literature focuses on the proportional asymptotics regime, where the number of features and samples diverge proportionally. Much of this work assumes i.i.d. Gaussian entries in the design. Concurrently, researchers have explored the universality of these findings, discovering that results based on the i.i.d. Gaussian assumption extend to other settings, including i.i.d. sub-Gaussian designs. However, universality results examining dependent covariates have predominanatly focused on correlation-based dependence or structured forms of dependence allowed by right-rotationally-invariant designs. In this paper, we challenge this limitation by investigating dependence structures beyond these established classes. We identify a class of designs characterized by a block dependence structure where results based on i.i.d. Gaussian designs persist. Formally, we establish that the optimal values of regularized empirical risk and the risk associated with convex regularized estimators, such as the Lasso and the ridge, converge to the same limit under block-dependent designs as for i.i.d. Gaussian entry designs. Our dependence structure differs significantly from correlation-based dependence and enables, for the first time, asymptotically exact risk characterization in prevalent high-dimensional nonlinear regression problems.