A Theory of the NEPv Approach for Optimization on the Stiefel Manifold

Abstract

The NEPv approach has been increasingly used lately for optimization on the Stiefel manifold arising from machine learning. General speaking, the approach first turns the first order optimality condition into a nonlinear eigenvalue problem with eigenvector dependency (NEPv) and then solve the nonlinear problem via some variations of the self-consistent-field (SCF) iteration. The difficulty, however, lies in designing a proper SCF iteration so that a maximizer is found at the end. Currently, each use of the approach is very much individualized, especially in its convergence analysis phase to show that the approach does work or otherwise. Related, the NPDo approach is recently proposed for the sum of coupled traces and it seeks to turn the first order optimality condition into a nonlinear polar decomposition with orthogonal factor dependency (NPDo). In this paper, two unifying frameworks are established, one for each approach. Each framework is built upon a basic assumption, under which globally convergence to a stationary point is guaranteed and during the SCF iterative process that leads to the stationary point, the objective function increases monotonically. Also the notion of atomic function for each approach is proposed, and the atomic functions include commonly used matrix traces of linear and quadratic forms as special ones. It is shown that the basic assumptions of the approaches are satisfied by their respective atomic functions and, more importantly, by convex compositions of their respective atomic functions. Together they provide a large collection of objectives for which either one of approaches or both are guaranteed to work, respectively.