Tractability from overparametrization: the example of the negative perceptron

Abstract

In the negative perceptron problem we are given n data points \((\varvec{x}_i,y_i)\), where \(\varvec{x}_i\) is a d-dimensional vector and \(y_i\in \{+1,-1\}\) is a binary label. The data are not linearly separable and hence we content ourselves to find a linear classifier with the largest possible negative margin. In other words, we want to find a unit norm vector \(\varvec{\theta }\) that maximizes \(\min _{i\le n}y_i\langle \varvec{\theta },\varvec{x}_i\rangle \). This is a non-convex optimization problem (it is equivalent to finding a maximum norm vector in a polytope), and we study its typical properties under two random models for the data. We consider the proportional asymptotics in which \(n,d\rightarrow \infty \) with \(n/d\rightarrow \delta \), and prove upper and lower bounds on the maximum margin \(\kappa _{{\textrm{s}}}(\delta )\) or—equivalently—on its inverse function \(\delta _{{\textrm{s}}}(\kappa )\). In other words, \(\delta _{{\textrm{s}}}(\kappa )\) is the overparametrization threshold: for \(n/d\le \delta _{{\textrm{s}}}(\kappa )-{\varepsilon }\) a classifier achieving vanishing training error exists with high probability, while for \(n/d\ge \delta _{{\textrm{s}}}(\kappa )+{\varepsilon }\) it does not. Our bounds on \(\delta _{{\textrm{s}}}(\kappa )\) match to the leading order as \(\kappa \rightarrow -\infty \). We then analyze a linear programming algorithm to find a solution, and characterize the corresponding threshold \(\delta _{\textrm{lin}}(\kappa )\). We observe a gap between the interpolation threshold \(\delta _{{\textrm{s}}}(\kappa )\) and the linear programming threshold \(\delta _{\textrm{lin}}(\kappa )\), raising the question of the behavior of other algorithms.