Speeding up the convergence of the Polyak’s Heavy Ball algorithm

Abstract

In the presented work, some procedures, usually used in modern algorithms of unconstrained optimization, are added to Polyak’s heavy ball method. Namely, periodical restarts, which guarantees monotonic decrease of the objective function along successive iterates, while restarts involve updating of the step size on the base of line search method.

For smooth objective functions, the Heavy Ball (briefly HB) and Modified Heavy Ball (briefly MHB) algorithms are described along with the problem of simplifying the form of used line-search algorithm (without changing its content). MHB and the set of test functions are implemented in C++. The set of test functions contains 44 functions, taken from Cuter/st. Solver CG_DESCENT-C-6.8 was used for MHB benchmarking. Test-functions and other materials, related to benchmarking, are uploaded to GitHub: https://github.com/kobage/.

In case of smooth and convex objective function, the convergence analysis is concentrated on reducing transformations and their orbits. A concept of reducing transformation allows us to investigate algebraic structure of convergent methods.