Explainable Gaussian Processes: a loss landscape perspective

Abstract

Prior beliefs about the latent function to shape inductive biases can be incorporated into a Gaussian Process (GP) via the kernel. However, beyond kernel choices, the decision-making process of GP models remains poorly understood. In this work, we contribute an analysis of the loss landscape for GP models using methods from chemical physics. We demonstrate $\nu$-continuity for Mat'ern kernels and outline aspects of catastrophe theory at critical points in the loss landscape. By directly including $\nu$ in the hyperparameter optimisation for Mat'ern kernels, we find that typical values of $\nu$ \textcolor{black}{can be} far from optimal in terms of performance. We also provide an \textit{a priori} method for evaluating the effect of GP ensembles and discuss various voting approaches based on physical properties of the loss landscape. The utility of these approaches is demonstrated for various synthetic and real datasets. Our findings provide \textcolor{black}{insight into hyperparameter optimisation for} GPs and offer practical guidance for improving their performance and interpretability in a range of applications.