A Hybrid Sampling Methodology Based Cost Effective Active Learning for Tabular Data

Abstract

Active learning (AL) is an attractive paradigm of machine learning that could be very useful in real-world machine learning for domains with high labelling costs. The main objective of AL is to efficiently acquire data for annotation from a typically large pool of unlabeled data, thereby reducing human experts' labelling effort. This helps them focus on the most informative data points that would improve machine learning model performance for the task at hand. Thus, given an unlabeled dataset and a fixed labelling budget, AL aims to select a subset of examples to be labelled such that they can result in improved model performance. The central component in an active learning workflow is the sampling process in which the most valuable samples to be labelled are identified. The most common type of sampling strategy employed is uncertainty sampling. In this sampling, each AL training iteration takes those points that the model is most uncertain about. Another important sampling strategy is diversity sampling. This sampling strategy selects a collection of samples that can well represent the entire data distribution. In this paper, we propose a hybrid sampling based active learning for the classification tasks employing tabular data, wherein we jointly sample uncertain as well as diverse points at every AL iteration. This hybrid sampling is achieved by jointly training a classification model along with an outlier detection model. In addition, we propose an improved cost-effective active learning (CEAL) method in which we automatically select high confidence data points and assign pseudo-labels based on not only the model's confidence but also the outlier detection module.