Rethinking density ratio estimation based hyper-parameter optimization

Abstract

Hyper-parameter optimization (HPO) aims to improve the performance of machine learning algorithms by identifying appropriate hyper-parameters. By converting the computation of expected improvement into density-ratio estimation problems, existing works use binary classifiers to estimate these ratio and determine the next point by maximizing the class posterior probabilities. However, these methods tend to treat different points equally and ignore some important regions, because binary classifiers are unable to capture more information about search spaces and highlight important regions. In this work, we propose a hyper-parameter optimization method by estimating ratios and selecting the next point using multi-class classifiers. First, we divide all samples into multiple classes and train multi-class classifiers. The decision boundaries of the trained classifiers allow for a finer partitioning of search spaces, offering richer insights into the distribution of hyper-parameters within search spaces. We then define an acquisition function as a weighted sum of multi-class classifiers’ outputs, with these weights determined by samples in each class. By assigning different weights to each class posterior probability in our acquisition function, points within search spaces are no longer treated equally. Experimental results on three representative tasks demonstrate that our method achieves a significant improvement in immediate regrets and convergence speed.