Integration of adaptive projection BFGS and inertial extrapolation step for nonconvex optimization problems and its application in machine learning

Abstract

With the rapid development of machine learning and large data technologies, large-scale optimization problems become more and more common, and traditional optimization algorithms face the challenges of computational complexity and memory consumption. In this paper, we combine the adaptive projection BFGS method with the inertial extrapolation technique, and propose a BFGS algorithm combining inertial extrapolation and adaptive projection, which enhances the search capability of the algorithm, ensures that there is a suitable descent direction in each iteration, and reduces the oscillation in the iteration. Under certain mild conditions and weak Wolfe–Powell (WWP) conditions, the algorithm demonstrates both global convergence and superlinear convergence rates for nonconvex unconstrained optimization problems. In addition, to solve large-scale optimization problems, we propose an inertial extrapolation adaptive projection BFGS algorithm based on variance reduction, which performs well in dealing with large-scale datasets and offers new insights to address the limitations of traditional algorithms. Finally, we evaluate their performance and efficiency using classical numerical experiments and applications in machine learning models.