Identifying unpredictable test examples with worst-case guarantees

Often times, whether it be for adversarial or natural reasons, the distributions of test and training data differ. We give an algorithm that, given sets of training and test examples, identifies regions of test examples that cannot be predicted with low error. These regions are classified as ƒ or equivalently omitted from classification. Assuming only that labels are consistent with a family of classifiers of low VC dimension, the algorithm is shown to make few misclassification errors and few errors of omission in both adversarial and covariate-shift settings. Previous models of learning with different training and test distributions required assumptions connecting the two.