Binary classification with covariate selection through ℓ0-penalised empirical risk minimisation

Abstract

We consider the problem of binary classification with covariate selection. We construct a classification procedure by minimising the empirical misclassification risk with a penalty on the number of selected covariates. This optimisation problem is equivalent to obtaining an ℓ0-penalised maximum score estimator. We derive probability bounds on the estimated sparsity as well as on the excess misclassification risk. These theoretical results are nonasymptotic and established in a high-dimensional setting. In particular, we show that our method yields a sparse solution whose ℓ0-norm can be arbitrarily close to true sparsity with high probability and obtain the rates of convergence for the excess misclassification risk. We implement the proposed procedure via the method of mixed-integer linear programming. Its numerical performance is illustrated in Monte Carlo experiments and a real data application of the work-trip transportation mode choice.