In this paper, we attempt to compare two distinct branches of research on second-order optimization methods. The first one studies self-concordant functions and barriers, the main assumption being that the third derivative of the objective is bounded by the second derivative. The second branch studies cubic regularized Newton methods (CRNMs) with the main assumption that the second derivative is Lipschitz continuous. We develop a new theoretical analysis for a path-following scheme (PFS) for general self-concordant functions, as opposed to the classical path-following scheme developed for self-concordant barriers. We show that the complexity bound for this scheme is better than that of the Damped Newton Method (DNM) and show that our method has global superlinear convergence. We propose also a new predictor-corrector path-following scheme (PCPFS) that leads to further improvement of constant factors in the complexity guarantees for minimizing general self-concordant functions. We also apply path-following schemes to different classes of constrained optimization problems and obtain the resulting complexity bounds. Finally, we analyze an important subclass of general self-concordant functions, namely a class of strongly convex functions with Lipschitz continuous second derivative, and show that for this subclass CRNMs give even better complexity bounds.
We develop a theory of finite-dimensional polyhedral subsets over the Wasserstein space and optimization of functionals over them via first-order methods. Our main application is to the problem of mean-field variational inference (MFVI), which seeks to approximate a distribution pi over mathbb {R}^d by a product measure pi ^star . When pi is strongly log-concave and log-smooth, we provide (1) approximation rates certifying that pi ^star is close to the minimizer pi ^star _diamond of the KL divergence over a polyhedral set mathcal {P}_diamond , and (2) an algorithm for minimizing mathop {textrm{KL}}limits (cdot !;Vert ; !pi ) over mathcal {P}_diamond based on accelerated gradient descent over mathbb {R}^d. As a byproduct of our analysis, we obtain the first end-to-end analysis for gradient-based algorithms for MFVI.
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：