Bayesian Optimization: Model Comparison With Different Benchmark Functions

Bayesian optimization(BO) is a global optimization problem. It is an important approach in machine learning, hyperparameter tuning and other fields such as drug discovery. BO consists of two main parts which are probabilistic model for the objective function and acquisition function. This paper mainly focused on assessing the strengths and weaknesses of two different probabilistic models which are Gaussian Process (GP) and Random Forests (RF). This paper illustrated several results, which indicated the performance of each probabilistic model and helped us find the optimal model corresponding to each benchmark function. RF will be preferred if the function is smooth. GP will be preferred if the function has many local minima. Moreover, implementability of other probabilistic models were discussed in this paper.