Descent Properties of an Anderson Accelerated Gradient Method with Restarting

Abstract

SIAM Journal on Optimization, Volume 34, Issue 1, Page 336-365, March 2024.
Abstract. Anderson acceleration ([math]) is a popular acceleration technique to enhance the convergence of fixed-point schemes. The analysis of [math] approaches often focuses on the convergence behavior of a corresponding fixed-point residual, while the behavior of the underlying objective function values along the accelerated iterates is currently not well understood. In this paper, we investigate local properties of [math] with restarting applied to a basic gradient scheme ([math]) in terms of function values. Specifically, we show that [math] is a local descent method and that it can decrease the objective function at a rate no slower than the gradient method up to higher-order error terms. These new results theoretically support the good numerical performance of [math] when heuristic descent conditions are used for globalization and they provide a novel perspective on the convergence analysis of [math] that is more amenable to nonconvex optimization problems. Numerical experiments are conducted to illustrate our theoretical findings.