Optimal Covariance Estimation for Condition Number Loss in the Spiked model

Abstract

Consider estimation of the covariance matrix under relative condition number loss , where is the condition number of matrix , and and are the estimated and theoretical covariance matrices. Recent advances in understanding the so-called for , are used here to derive a nonlinear shrinker which is asymptotically optimal among orthogonally-covariant procedures. These advances apply in an asymptotic setting, where the number of variables is comparable to the number of observations . The form of the optimal nonlinearity depends on the aspect ratio of the data matrix and on the top eigenvalue of . For , even dependence on the top eigenvalue can be avoided. The optimal shrinker has three notable properties. First, when is moderately large, it shrinks even very large eigenvalues substantially, by a factor . Second, even for moderate , certain highly statistically significant eigencomponents will be completely suppressed.Third, when is very large, the optimal covariance estimator can be purely diagonal, despite the top theoretical eigenvalue being large and the empirical eigenvalues being highly statistically significant. This aligns with practitioner experience. Alternatively, certain non-optimal intuitively reasonable procedures can have small worst-case relative regret - the simplest being generalized soft thresholding having threshold at the bulk edge and slope above the bulk. For this has at most a few percent relative regret.