Metaheuristic optimisation of Gaussian process regression model hyperparameters: Insights from FEREBUS

Abstract

FEREBUS is a Gaussian process regression (GPR) engine embedded in the large machinery of FFLUX, a novel machine learnt force field developed from scratch through several well-documented proof-of-concept studies. This package relies on the exploration and exploitation capabilities of metaheuristic algorithms (MAs) to carry out the global optimisation of GPR model hyperparameters ($\theta$). However, because MAs employ different search mechanisms to scrutinise the hyperparameter space, their performance on a specific optimisation task can vary a lot from one technique to another. Herein, we report a series of carefully designed experiments aimed at evaluating the ability of ten metaheuristic algorithms to locate the optimal set of $\theta$ values. Selected optimisation techniques belong to four popular families of MAs, namely particle swarm optimisation (4), grey wolf optimisation (2), bat (2) and firefly (2) algorithms. Our calculations suggest that grey wolf optimisers (GWOs) achieve the best results on average. Furthermore, the RMSE($\theta$) cost function is confirmed to be an excellent guide for the selection of atomic GPR models. This work also briefly introduces an enhanced grey wolf optimiser called GWO-RUHL (Random Update of the Hierarchy Ladder), which accounts for the (so far omitted) natural desire of non-leader wolves to occupy high-ranked leadership positions in the pack. We demonstrate that GWO-RUHL achieves better results than the standard GWO in terms of both convergence speed and quality of solutions.