A new property of convex functions that makes it possible to achieve the linear rate of convergence of the Newton method during the minimization process is established. Namely, it is proved that, even in the case of singularity of the Hessian at the solution, the Newtonian system is solvable in the vicinity of the minimizer; i.e., the gradient of the objective function belongs to the image of the matrix of second derivatives and, therefore, analogs of the Newton method may be used.
Lyapunov functions play a fundamental role in analyzing the stability and convergence properties of optimization methods. In this paper, we propose a novel and straightforward approach for constructing Lyapunov functions for first-order methods applied to quadratic functions. Our approach involves bringing the iteration matrix to an upper triangular form using Schur decomposition, then examining the value of the last coordinate of the state vector. This value is multiplied by a magnitude smaller than one at each iteration. Consequently, this value should decrease at each iteration, provided that the method converges. We rigorously prove the suitability of this Lyapunov function for all first-order methods and derive the necessary conditions for the proposed function to decrease monotonically. Experiments conducted with general convex functions are also presented, alongside a study on the limitations of the proposed approach. Remarkably, the newly discovered L-yapunov function is straightforward and does not explicitly depend on the exact method formulation or function characteristics like strong convexity or smoothness constants. In essence, a single expression serves as a Lyapunov function for several methods, including Heavy Ball, Nesterov Accelerated Gradient, and Triple Momentum, among others. To the best of our knowledge, this approach has not been previously reported in the literature.
The study of nonlinear problems related to heat transfer in a substance is of great practical important. Earlier, this paper’s authors proposed an effective algorithm for determining the volumetric heat capacity and thermal conductivity of a substance based on experimental observations of the dynamics of the temperature field in the object. In this paper, the problem of simultaneous identification of temperature-dependent volumetric heat capacity and thermal conductivity of the substance under study from the heat flux at the boundary of the domain is investigated. The consideration is based on the first (Dirichlet) boundary value problem for a one-dimensional unsteady heat equation. The coefficient inverse problem under consideration is reduced to a variational problem, which is solved by gradient methods based on the application of fast automatic differentiation. The uniqueness of the solution of the inverse problem is investigated.
The history of the alternating projection method for finding a common point of several convex sets in Euclidean space goes back to the well-known Kaczmarz algorithm for solving systems of linear equations, which was devised in the 1930s and later found wide applications in image processing and computed tomography. An important role in the study of this method was played by I.I. Eremin’s, L.M. Bregman’s, and B.T. Polyak’s works, which appeared nearly simultaneously and contained general results concerning the convergence of alternating projections to a point in the intersection of sets, assuming that this intersection is nonempty. In this paper, we consider a modification of the convex set intersection problem that is related to the theory of multi-agent systems and is called the constrained consensus problem. Each convex set in this problem is associated with a certain agent and, generally speaking, is inaccessible to the other agents. A group of agents is interested in finding a common point of these sets, that is, a point satisfying all the constraints. Distributed analogues of the alternating projection method proposed for solving this problem lead to a rather complicated nonlinear system of equations, the convergence of which is usually proved using special Lyapunov functions. A brief survey of these methods is given, and their relation to the theorem ensuring consensus in a system of averaging inequalities recently proved by the second author is shown (this theorem develops convergence results for the usual method of iterative averaging as applied to the consensus problem).
Stochastic Gradient Descent (SGD) is one of the many iterative optimization methods that are widely used in solving machine learning problems. These methods display valuable properties and attract researchers and industrial machine learning engineers with their simplicity. However, one of the weaknesses of this type of methods is the necessity to tune learning rate (step-size) for every loss function and dataset combination to solve an optimization problem and get an efficient performance in a given time budget. Stochastic Gradient Descent with Polyak Step-size (SPS) is a method that offers an update rule that alleviates the need of fine-tuning the learning rate of an optimizer. In this paper, we propose an extension of SPS that employs preconditioning techniques, such as Hutchinson’s method, Adam, and AdaGrad, to improve its performance on badly scaled and/or ill-conditioned datasets.
A new method is introduced for the construction of control variates to reduce the variance of additive functionals of Markov Chain Monte Carlo (MCMC) samplers. These control variates are obtained by minimizing the asymptotic variance associated with the Langevin diffusion over a family of functions. To motivate our approach, we then show that the asymptotic variance of some well-known MCMC algorithms, including the Random Walk Metropolis and the (Metropolis) Unadjusted/Adjusted Langevin Algorithm, are well approximated by that of the Langevin diffusion. We finally theoretically justify the use of a class of linear control variates we introduce. In particular, we show that the variance of the resulting estimators is smaller, for a given computational complexity, than the standard Monte Carlo estimator. Several examples of Bayesian inference problems support our findings showing, in some cases, very significant reduction of the variance.
Iterative learning control (ILC) algorithms appeared in connection with the problems of increasing the accuracy of performing repetitive operations by robots. They use information from previous repetitions to adjust the control signal on the current repetition. Most often, information from the previous repetition only is used. ILC algorithms that use information from several previous iterations are called higher-order algorithms. Recently, interest in these algorithms has increased in the literature in connection with robotic additive manufacturing problems. However, in addition to the fact that these algorithms have been little studied, there are conflicting estimates regarding their properties. This paper proposes new higher-order ILC algorithms for linear discrete and differential systems. The idea of these algorithms is based on an analogy with multi-step methods in optimization theory, in particular, with the heavy ball method. An example is given that confirms the possibility to accelerate convergence of the learning error when using such algorithms.
In this paper, we study the black box optimization problem under the Polyak–Lojasiewicz (PL) condition, assuming that the objective function is not just smooth, but has higher smoothness. By using “kernel-based” approximations instead of the exact gradient in the Stochastic Gradient Descent method, we improve the best-known results of convergence in the class of gradient-free algorithms solving problems under the PL condition. We generalize our results to the case where a zeroth-order oracle returns a function value at a point with some adversarial noise. We verify our theoretical results on the example of solving a system of nonlinear equations.
The joint numerical range of tuples of matrices is a powerful tool for proving results which are useful in optimization, such as the (mathcal{S})-lemma. Here we provide a similar proof for another result, namely the equivalence of a certain positivity criterion to Duffin’s overdamping condition involving quadratic matrix-valued polynomials. We show how the proof is generalizable to higher degrees of matrix-valued polynomials.
We consider minimization of the supporting function of a convex compact set on the unit sphere. In essence, this is the problem of projecting zero onto a compact convex set. We consider sufficient conditions for solving this problem with a linear rate using a first order algorithm—the gradient projection method with a fixed step-size and with Armijo’s step-size. We consider some applications for problems with set-valued mappings. The mappings in the work basically are given through the set-valued integral of a set-valued mapping with convex and compact images or as the Minkowski sum of finite number of convex compact sets, e.g., ellipsoids. Unlike another solution ways, e.g., with approximation in a certain sense of the mapping, the considered algorithm much weaker depends on the dimension of the space and other parameters of the problem. It also allows efficient error estimation. Numerical experiments confirm the effectiveness of the considered approach.
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：