Generalized Hessian approximations via Stein's lemma for constrained minimization

Abstract

We consider the problem of convex constrained minimization of an average of n functions, where the parameter and the features are related through inner products. We focus on second order batch updates, where the curvature matrix is obtained by assuming random design and by applying the celebrated Stein's lemma together with subsampling techniques. The proposed algorithm enjoys fast convergence rates similar to the Newton method, yet the per-iteration cost has the same order of magnitude as the gradient descent. We demonstrate its performance on well-known optimization problems where Stein's lemma is not directly applicable, such as M-estimation for robust statistics, and inequality form linear/quadratic programming etc. Under certain assumptions, we show that the constrained optimization algorithm attains a composite convergence rate that is initially quadratic and asymptotically linear. We validate its performance through widely encountered optimization tasks on several real and synthetic datasets by comparing it to classical optimization algorithms.