Learning Curve Estimation with Large Imbalanced Datasets

Abstract

Datasets for machine learning are constantly increasing in size, along with computational requirements for processing the data. A useful exercise for machine learning experiments is to approximate model performance as dataset size increases. This can inform application building and data collection efforts as well as improve computational efficiency by using subsets of the data. In this paper, we evaluate a learning curve estimation method on three large imbalanced datasets. Estimation is performed by fitting an inverse power law model to a learning curve created on a small amount of data. We then explore how well this estimated curve fits to the full learning curve of each dataset. The method has been previously evaluated for small datasets (hundreds or thousands of instances), and in this study we show that the method is indeed effective for larger datasets with millions of instances. This is beneficial because only a few thousand instances are required to accurately estimate the performance of models using millions of instances. To the best of our knowledge, this is the first study to systematically explore the use of an inverse power law curve fitting method for big data.