Unified Classical and Robust Optimization for Least Squares

The solutions to robust optimization problems are sometimes too conservative because of the focus on worst-case performance. For the least-squares (LS) problem, we describe a way to overcome this by combining the classical formulation with its robust version. We do this by constructing a sequence of problems that are parameterized in terms of the well-estimated aspects of the data. One end of this sequence is the Classical LS, and the other end is a variant of Robust LS that we construct for this purpose. By choosing the right point in the sequence, we are selectively robust only to the poorly estimated aspects of the data. However, we show that better estimation does not imply better prediction. We then transform the problem to align the estimation and prediction objectives, calling it objective matching. This transformation improves prediction while provably retaining the problem structure. Objective matching allows our method (called Unified Least Squares or ULS) to consistently match or outperform other state-of-the-art techniques, including both ridge and LASSO regression, on simulations and real-world data sets.