A Robust Stochastic Quasi-Newton Method with the Application in Machine Learning

Abstract

We study a novel stochastic version of damped and regularized limited memory Broyden-Fletcher-Goldfarb-Shanno (Sd-REG-LBFGS) method for nonconvex optimization problems in this paper. Different from BFGS updating scheme for the strongly convex problems, it is a challenge to preserve the product of the correction pairs positive at each iteration for BFGS in nonconvex case. While utilizing regularization scheme to ensure the robustness of the second-order methods. The proposed method is able to keep the product of correction pairs being away from zero above a specified degree. To make the proposed method robust and computationally efficient, we propose to update curvature information at a spaced interval, in which the average of iterate points is utilized. The numerical experiments have shown that our proposed algorithm has better performance than SdLBFGS or they have almost the same performance. Especially in the problems which utilizes few samples, the proposed method has avoided the ill-conditioned problem and exhibited superior performance.