Numerical properties of solutions of LASSO regression

Abstract

The determination of a concise model of a linear system when there are fewer samples m than predictors n requires the solution of the equation $Ax=b$, where $A\in R^{m\times n}$ and $\text{rank}A=m$, such that the selected solution from the infinite number of solutions is sparse, that is, many of its components are zero. This leads to the minimisation with respect to x of $f(x,\lambda )={\Vert Ax-b\Vert }_2^2+\lambda {\Vert x\Vert }_1$, where λ is the regularisation parameter. This problem, which is called LASSO regression, yields a family of functions $x_{\text{lasso}}\left(\lambda \right)$ and it is necessary to determine the optimal value of λ, that is, the value of λ that balances the fidelity of the model, $\Vert A{x}_{\text{lasso}}\left(\lambda \right)-b\Vert \approx 0$, and the satisfaction of the constraint that $x_{\text{lasso}}\left(\lambda \right)$ be sparse. The aim of this paper is an investigation of the numerical properties of $x_{\text{lasso}}\left(\lambda \right)$, and the main conclusion of this investigation is the incompatibility of sparsity and stability, that is, a sparse solution $x_{\text{lasso}}\left(\lambda \right)$ that preserves the fidelity of the model exists if the least squares (LS) solution $x_{\text{ls}}=A^{†}b$ is unstable. Two methods, cross validation and the L-curve, for the computation of the optimal value of λ are compared and it is shown that the L-curve yields significantly better results. This difference between stable and unstable solutions $x_{\text{ls}}$ of the LS problem manifests itself in the very different forms of the L-curve for these two solutions. The paper includes examples of stable and unstable solutions $x_{\text{ls}}$ that demonstrate the theory.