SIAM Journal on Optimization, Volume 34, Issue 3, Page 2588-2608, September 2024. Abstract. This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster [math] rate for gradient descent is also motivated along with simple numerical validation.
{"title":"Provably Faster Gradient Descent via Long Steps","authors":"Benjamin Grimmer","doi":"10.1137/23m1588408","DOIUrl":"https://doi.org/10.1137/23m1588408","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2588-2608, September 2024. <br/> Abstract. This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster [math] rate for gradient descent is also motivated along with simple numerical validation.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141745956","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lien T. Nguyen, Andrew Eberhard, Xinghuo Yu, Chaojie Li
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2557-2587, September 2024. Abstract. In this paper, we propose a fast gradient algorithm for the problem of minimizing a differentiable (possibly nonconvex) function in Hilbert spaces. We first extend the dry friction property for convex functions to what we call the dry-like friction property in a nonconvex setting, and then employ a line search technique to adaptively update parameters at each iteration. Depending on the choice of parameters, the proposed algorithm exhibits subsequential convergence to a critical point or full sequential convergence to an “approximate” critical point of the objective function. We also establish the full sequential convergence to a critical point under the Kurdyka–Łojasiewicz (KL) property of a merit function. Thanks to the parameters’ flexibility, our algorithm can reduce to a number of existing inertial gradient algorithms with Hessian damping and dry friction. By exploiting variational properties of the Moreau envelope, the proposed algorithm is adapted to address weakly convex nonsmooth optimization problems. In particular, we extend the result on KL exponent for the Moreau envelope of a convex KL function to a broad class of KL functions that are not necessarily convex nor continuous. Simulation results illustrate the efficiency of our algorithm and demonstrate the potential advantages of combining dry-like friction with extrapolation and line search techniques.
{"title":"Fast Gradient Algorithm with Dry-like Friction and Nonmonotone Line Search for Nonconvex Optimization Problems","authors":"Lien T. Nguyen, Andrew Eberhard, Xinghuo Yu, Chaojie Li","doi":"10.1137/22m1532354","DOIUrl":"https://doi.org/10.1137/22m1532354","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2557-2587, September 2024. <br/> Abstract. In this paper, we propose a fast gradient algorithm for the problem of minimizing a differentiable (possibly nonconvex) function in Hilbert spaces. We first extend the dry friction property for convex functions to what we call the dry-like friction property in a nonconvex setting, and then employ a line search technique to adaptively update parameters at each iteration. Depending on the choice of parameters, the proposed algorithm exhibits subsequential convergence to a critical point or full sequential convergence to an “approximate” critical point of the objective function. We also establish the full sequential convergence to a critical point under the Kurdyka–Łojasiewicz (KL) property of a merit function. Thanks to the parameters’ flexibility, our algorithm can reduce to a number of existing inertial gradient algorithms with Hessian damping and dry friction. By exploiting variational properties of the Moreau envelope, the proposed algorithm is adapted to address weakly convex nonsmooth optimization problems. In particular, we extend the result on KL exponent for the Moreau envelope of a convex KL function to a broad class of KL functions that are not necessarily convex nor continuous. Simulation results illustrate the efficiency of our algorithm and demonstrate the potential advantages of combining dry-like friction with extrapolation and line search techniques.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141719096","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Roger Behling, Yunier Bello-Cruz, Alfredo N. Iusem, Di Liu, Luiz-Rafael Santos
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2535-2556, September 2024. Abstract. In this paper, we present a variant of the circumcenter method for the convex feasibility problem (CFP), ensuring finite convergence under a Slater assumption. The method replaces exact projections onto the convex sets with projections onto separating half-spaces, perturbed by positive exogenous parameters that decrease to zero along the iterations. If the perturbation parameters decrease slowly enough, such as the terms of a diverging series, finite convergence is achieved. To the best of our knowledge, this is the first circumcenter method for CFP that guarantees finite convergence.
{"title":"A Finitely Convergent Circumcenter Method for the Convex Feasibility Problem","authors":"Roger Behling, Yunier Bello-Cruz, Alfredo N. Iusem, Di Liu, Luiz-Rafael Santos","doi":"10.1137/23m1595412","DOIUrl":"https://doi.org/10.1137/23m1595412","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2535-2556, September 2024. <br/> Abstract. In this paper, we present a variant of the circumcenter method for the convex feasibility problem (CFP), ensuring finite convergence under a Slater assumption. The method replaces exact projections onto the convex sets with projections onto separating half-spaces, perturbed by positive exogenous parameters that decrease to zero along the iterations. If the perturbation parameters decrease slowly enough, such as the terms of a diverging series, finite convergence is achieved. To the best of our knowledge, this is the first circumcenter method for CFP that guarantees finite convergence.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141719097","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2503-2534, September 2024. Abstract. The Frank–Wolfe method has become increasingly useful in statistical and machine learning applications due to the structure-inducing properties of the iterates and especially in settings where linear minimization over the feasible set is more computationally efficient than projection. In the setting of empirical risk minimization—one of the fundamental optimization problems in statistical and machine learning—the computational effectiveness of Frank–Wolfe methods typically grows linearly in the number of data observations [math]. This is in stark contrast to the case for typical stochastic projection methods. In order to reduce this dependence on [math], we look to second-order smoothness of typical smooth loss functions (least squares loss and logistic loss, for example), and we propose amending the Frank–Wolfe method with Taylor series–approximated gradients, including variants for both deterministic and stochastic settings. Compared with current state-of-the-art methods in the regime where the optimality tolerance [math] is sufficiently small, our methods are able to simultaneously reduce the dependence on large [math] while obtaining optimal convergence rates of Frank–Wolfe methods in both convex and nonconvex settings. We also propose a novel adaptive step-size approach for which we have computational guarantees. Finally, we present computational experiments which show that our methods exhibit very significant speedups over existing methods on real-world datasets for both convex and nonconvex binary classification problems.
{"title":"Using Taylor-Approximated Gradients to Improve the Frank–Wolfe Method for Empirical Risk Minimization","authors":"Zikai Xiong, Robert M. Freund","doi":"10.1137/22m1519286","DOIUrl":"https://doi.org/10.1137/22m1519286","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2503-2534, September 2024. <br/> Abstract. The Frank–Wolfe method has become increasingly useful in statistical and machine learning applications due to the structure-inducing properties of the iterates and especially in settings where linear minimization over the feasible set is more computationally efficient than projection. In the setting of empirical risk minimization—one of the fundamental optimization problems in statistical and machine learning—the computational effectiveness of Frank–Wolfe methods typically grows linearly in the number of data observations [math]. This is in stark contrast to the case for typical stochastic projection methods. In order to reduce this dependence on [math], we look to second-order smoothness of typical smooth loss functions (least squares loss and logistic loss, for example), and we propose amending the Frank–Wolfe method with Taylor series–approximated gradients, including variants for both deterministic and stochastic settings. Compared with current state-of-the-art methods in the regime where the optimality tolerance [math] is sufficiently small, our methods are able to simultaneously reduce the dependence on large [math] while obtaining optimal convergence rates of Frank–Wolfe methods in both convex and nonconvex settings. We also propose a novel adaptive step-size approach for which we have computational guarantees. Finally, we present computational experiments which show that our methods exhibit very significant speedups over existing methods on real-world datasets for both convex and nonconvex binary classification problems.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141613419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2472-2502, September 2024. Abstract. In this paper, we present complexity analysis of proximal inexact gradient methods for finite-sum optimization with a nonconvex nonsmooth composite function and non-Lipschitz regularization. By getting access to a convex approximation to the Lipschitz function and a Lipschitz continuous approximation to the non-Lipschitz regularizer, we construct a proximal subproblem at each iteration without using exact function values and gradients. With certain accuracy control on inexact gradients and subproblem solutions, we show that the oracle complexity in terms of total number of inexact gradient evaluations is in order [math] to find an [math]-approximate first-order stationary point, ensuring that within a [math]-ball centered at this point the maximum reduction of an approximation model does not exceed [math]. This shows that we can have the same worst-case evaluation complexity order as in [C. Cartis, N. I. M. Gould, and P. L. Toint, SIAM J. Optim., 21 (2011), pp. 1721–1739, X. Chen, Ph. L. Toint, and H. Wang, SIAM J. Optim., 29 (2019), pp. 874–903], even if we introduce the non-Lipschitz singularity and the nonconvex nonsmooth composite function in the objective function. Moreover, we establish that the oracle complexity regarding the total number of stochastic oracles is in order [math] with high probability for stochastic proximal inexact gradient methods. We further extend the algorithm to adjust to solving stochastic problems with expectation form and derive the associated oracle complexity in order [math] with high probability.
SIAM 优化期刊》，第 34 卷第 3 期，第 2472-2502 页，2024 年 9 月。 摘要本文提出了近似非精确梯度方法的复杂性分析，用于非凸非光滑复合函数和非 Lipschitz 正则化的有限和优化。通过获取 Lipschitz 函数的凸近似值和非 Lipschitz 正则化的 Lipschitz 连续近似值，我们可以在每次迭代时构建一个近似子问题，而无需使用精确的函数值和梯度。在对非精确梯度和子问题解进行一定精度控制的情况下，我们证明了以非精确梯度求值总数为单位的oracle复杂度是[math]，以找到一个[math]近似的一阶静止点，确保在以该点为中心的[math]球内，近似模型的最大还原度不超过[math]。这表明，我们可以获得与 [C. Cartis, N. I.] 中相同的最坏情况评估复杂度阶次。Cartis、N. I.M. Gould, and P. L. Toint, SIAM J. Optim., 21 (2011), pp.此外，我们还证明，对于随机近似不精确梯度法来说，关于随机神谕总数的神谕复杂度很有可能是[math]。我们进一步扩展了该算法，以适应求解期望形式的随机问题，并推导出相关的神谕复杂度以高概率为[math]阶。
{"title":"Complexity of Finite-Sum Optimization with Nonsmooth Composite Functions and Non-Lipschitz Regularization","authors":"Xiao Wang, Xiaojun Chen","doi":"10.1137/23m1546701","DOIUrl":"https://doi.org/10.1137/23m1546701","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2472-2502, September 2024. <br/> Abstract. In this paper, we present complexity analysis of proximal inexact gradient methods for finite-sum optimization with a nonconvex nonsmooth composite function and non-Lipschitz regularization. By getting access to a convex approximation to the Lipschitz function and a Lipschitz continuous approximation to the non-Lipschitz regularizer, we construct a proximal subproblem at each iteration without using exact function values and gradients. With certain accuracy control on inexact gradients and subproblem solutions, we show that the oracle complexity in terms of total number of inexact gradient evaluations is in order [math] to find an [math]-approximate first-order stationary point, ensuring that within a [math]-ball centered at this point the maximum reduction of an approximation model does not exceed [math]. This shows that we can have the same worst-case evaluation complexity order as in [C. Cartis, N. I. M. Gould, and P. L. Toint, SIAM J. Optim., 21 (2011), pp. 1721–1739, X. Chen, Ph. L. Toint, and H. Wang, SIAM J. Optim., 29 (2019), pp. 874–903], even if we introduce the non-Lipschitz singularity and the nonconvex nonsmooth composite function in the objective function. Moreover, we establish that the oracle complexity regarding the total number of stochastic oracles is in order [math] with high probability for stochastic proximal inexact gradient methods. We further extend the algorithm to adjust to solving stochastic problems with expectation form and derive the associated oracle complexity in order [math] with high probability.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141585848","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2440-2471, September 2024. Abstract. We examine the last-iterate convergence rate of Bregman proximal methods—from mirror descent to mirror-prox and its optimistic variants—as a function of the local geometry induced by the prox-mapping defining the method. For generality, we focus on local solutions of constrained, nonmonotone variational inequalities, and we show that the convergence rate of a given method depends sharply on its associated Legendre exponent, a notion that measures the growth rate of the underlying Bregman function (Euclidean, entropic, or other) near a solution. In particular, we show that boundary solutions exhibit a stark separation of regimes between methods with a zero and nonzero Legendre exponent: The former converge at a linear rate, while the latter converge, in general, sublinearly. This dichotomy becomes even more pronounced in linearly constrained problems where methods with entropic regularization achieve a linear convergence rate along sharp directions, compared to convergence in a finite number of steps under Euclidean regularization.
{"title":"The Rate of Convergence of Bregman Proximal Methods: Local Geometry Versus Regularity Versus Sharpness","authors":"Waïss Azizian, Franck Iutzeler, Jérôme Malick, Panayotis Mertikopoulos","doi":"10.1137/23m1580218","DOIUrl":"https://doi.org/10.1137/23m1580218","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2440-2471, September 2024. <br/> Abstract. We examine the last-iterate convergence rate of Bregman proximal methods—from mirror descent to mirror-prox and its optimistic variants—as a function of the local geometry induced by the prox-mapping defining the method. For generality, we focus on local solutions of constrained, nonmonotone variational inequalities, and we show that the convergence rate of a given method depends sharply on its associated Legendre exponent, a notion that measures the growth rate of the underlying Bregman function (Euclidean, entropic, or other) near a solution. In particular, we show that boundary solutions exhibit a stark separation of regimes between methods with a zero and nonzero Legendre exponent: The former converge at a linear rate, while the latter converge, in general, sublinearly. This dichotomy becomes even more pronounced in linearly constrained problems where methods with entropic regularization achieve a linear convergence rate along sharp directions, compared to convergence in a finite number of steps under Euclidean regularization.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141575207","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Guozhi Dong, Michael Hintermüller, Kostas Papafitsoros
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2314-2349, September 2024. Abstract. We propose and analyze a numerical algorithm for solving a class of optimal control problems for learning-informed semilinear partial differential equations (PDEs). Such PDEs contain constituents that are in principle unknown and are approximated by nonsmooth ReLU neural networks. We first show that direct smoothing of the ReLU network with the aim of using classical numerical solvers can have disadvantages, such as potentially introducing multiple solutions for the corresponding PDE. This motivates us to devise a numerical algorithm that treats directly the nonsmooth optimal control problem, by employing a descent algorithm inspired by a bundle-free method. Several numerical examples are provided and the efficiency of the algorithm is shown.
{"title":"A Descent Algorithm for the Optimal Control of ReLU Neural Network Informed PDEs Based on Approximate Directional Derivatives","authors":"Guozhi Dong, Michael Hintermüller, Kostas Papafitsoros","doi":"10.1137/22m1534420","DOIUrl":"https://doi.org/10.1137/22m1534420","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2314-2349, September 2024. <br/> Abstract. We propose and analyze a numerical algorithm for solving a class of optimal control problems for learning-informed semilinear partial differential equations (PDEs). Such PDEs contain constituents that are in principle unknown and are approximated by nonsmooth ReLU neural networks. We first show that direct smoothing of the ReLU network with the aim of using classical numerical solvers can have disadvantages, such as potentially introducing multiple solutions for the corresponding PDE. This motivates us to devise a numerical algorithm that treats directly the nonsmooth optimal control problem, by employing a descent algorithm inspired by a bundle-free method. Several numerical examples are provided and the efficiency of the algorithm is shown.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141519737","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2350-2377, September 2024. Abstract. We present new large-scale algorithms for fitting a subgradient regularized multivariate convex regression function to [math] samples in [math] dimensions—a key problem in shape constrained nonparametric regression with applications in statistics, engineering, and the applied sciences. The infinite-dimensional learning task can be expressed via a convex quadratic program (QP) with [math] decision variables and [math] constraints. While instances with [math] in the lower thousands can be addressed with current algorithms within reasonable runtimes, solving larger problems (e.g., [math] or [math]) is computationally challenging. To this end, we present an active set type algorithm on the dual QP. For computational scalability, we allow for approximate optimization of the reduced subproblems and propose randomized augmentation rules for expanding the active set. We derive novel computational guarantees for our algorithms. We demonstrate that our framework can approximately solve instances of the subgradient regularized convex regression problem with [math] and [math] within minutes and shows strong computational performance compared to earlier approaches.
{"title":"Subgradient Regularized Multivariate Convex Regression at Scale","authors":"Wenyu Chen, Rahul Mazumder","doi":"10.1137/21m1413134","DOIUrl":"https://doi.org/10.1137/21m1413134","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2350-2377, September 2024. <br/> Abstract. We present new large-scale algorithms for fitting a subgradient regularized multivariate convex regression function to [math] samples in [math] dimensions—a key problem in shape constrained nonparametric regression with applications in statistics, engineering, and the applied sciences. The infinite-dimensional learning task can be expressed via a convex quadratic program (QP) with [math] decision variables and [math] constraints. While instances with [math] in the lower thousands can be addressed with current algorithms within reasonable runtimes, solving larger problems (e.g., [math] or [math]) is computationally challenging. To this end, we present an active set type algorithm on the dual QP. For computational scalability, we allow for approximate optimization of the reduced subproblems and propose randomized augmentation rules for expanding the active set. We derive novel computational guarantees for our algorithms. We demonstrate that our framework can approximately solve instances of the subgradient regularized convex regression problem with [math] and [math] within minutes and shows strong computational performance compared to earlier approaches.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141519736","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2259-2286, September 2024. Abstract. We present a new gradient-like dynamical system related to unconstrained convex smooth multiobjective optimization which involves inertial effects and asymptotic vanishing damping. To the best of our knowledge, this system is the first inertial gradient-like system for multiobjective optimization problems including asymptotic vanishing damping, expanding the ideas previously laid out in [H. Attouch and G. Garrigos, Multiobjective Optimization: An Inertial Dynamical Approach to Pareto Optima, preprint, arXiv:1506.02823, 2015]. We prove existence of solutions to this system in finite dimensions and further prove that its bounded solutions converge weakly to weakly Pareto optimal points. In addition, we obtain a convergence rate of order [math] for the function values measured with a merit function. This approach presents a good basis for the development of fast gradient methods for multiobjective optimization.
SIAM 优化期刊》，第 34 卷第 3 期，第 2259-2286 页，2024 年 9 月。 摘要我们提出了一个与无约束凸平滑多目标优化相关的新的类梯度动力系统，它涉及惯性效应和渐近消失阻尼。据我们所知，该系统是第一个包含渐近消失阻尼的多目标优化问题的惯性类梯度系统，拓展了以前在 [H. Attouch and G. Garrigou] 中提出的观点。Attouch 和 G. Garrigos，多目标优化：An Inertial Dynamical Approach to Pareto Optima, preprint, arXiv:1506.02823, 2015]中的观点。我们证明了该系统在有限维度上的解的存在性，并进一步证明了其有界解弱收敛于弱帕累托最优点。此外，我们还获得了用绩优函数测量的函数值的[math]阶收敛率。这种方法为开发多目标优化的快速梯度方法奠定了良好基础。
{"title":"Fast Convergence of Inertial Multiobjective Gradient-Like Systems with Asymptotic Vanishing Damping","authors":"Konstantin Sonntag, Sebastian Peitz","doi":"10.1137/23m1588512","DOIUrl":"https://doi.org/10.1137/23m1588512","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2259-2286, September 2024. <br/> Abstract. We present a new gradient-like dynamical system related to unconstrained convex smooth multiobjective optimization which involves inertial effects and asymptotic vanishing damping. To the best of our knowledge, this system is the first inertial gradient-like system for multiobjective optimization problems including asymptotic vanishing damping, expanding the ideas previously laid out in [H. Attouch and G. Garrigos, Multiobjective Optimization: An Inertial Dynamical Approach to Pareto Optima, preprint, arXiv:1506.02823, 2015]. We prove existence of solutions to this system in finite dimensions and further prove that its bounded solutions converge weakly to weakly Pareto optimal points. In addition, we obtain a convergence rate of order [math] for the function values measured with a merit function. This approach presents a good basis for the development of fast gradient methods for multiobjective optimization.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141519781","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alejandro Carderera, Mathieu Besançon, Sebastian Pokutta
SIAM Journal on Optimization, Volume 34, Issue 3, Page 2231-2258, September 2024. Abstract. Generalized self-concordance is a key property present in the objective function of many important learning problems. We establish the convergence rate of a simple Frank–Wolfe variant that uses the open-loop step size strategy [math], obtaining an [math] convergence rate for this class of functions in terms of primal gap and Frank–Wolfe gap, where [math] is the iteration count. This avoids the use of second-order information or the need to estimate local smoothness parameters of previous work. We also show improved convergence rates for various common cases, e.g., when the feasible region under consideration is uniformly convex or polyhedral.
{"title":"Scalable Frank–Wolfe on Generalized Self-Concordant Functions via Simple Steps","authors":"Alejandro Carderera, Mathieu Besançon, Sebastian Pokutta","doi":"10.1137/23m1616789","DOIUrl":"https://doi.org/10.1137/23m1616789","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2231-2258, September 2024. <br/> Abstract. Generalized self-concordance is a key property present in the objective function of many important learning problems. We establish the convergence rate of a simple Frank–Wolfe variant that uses the open-loop step size strategy [math], obtaining an [math] convergence rate for this class of functions in terms of primal gap and Frank–Wolfe gap, where [math] is the iteration count. This avoids the use of second-order information or the need to estimate local smoothness parameters of previous work. We also show improved convergence rates for various common cases, e.g., when the feasible region under consideration is uniformly convex or polyhedral.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":null,"pages":null},"PeriodicalIF":3.1,"publicationDate":"2024-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141519782","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号