Kernel Factorisation Machines

Abstract

This paper explores a generalisation of factorisation machines via kernels, which we call Kernel Factorisation Machines (“KFM”). It is well-known that functions in reproducing kernel Hilbert spaces can be understood as a linear combination of features in very high-dimensional (or infinite-dimensional) spaces while being computed in a finite-dimensional space, thanks to the representer theorem. Simultaneously, it has been shown recently that the dot product operation was a key component behind the success of a number of recommender systems, while the recent literature has been preoccupied with enriching factorisation machines. There is thus a need for a framework able to interpolate between factorisation machines that tend to outperform other techniques on sparse datasets and more advanced models that perform well on large and dense datasets. One of the drawbacks of kernel methods is their high dimensionality when the number of observations is large, which is typical of recommender systems. It is thus extremely important to be able to reduce the dimensionality, which we do in two different ways: first, we find a representation of the input features in a lower-dimensional space, and, second, we consider inducing points, i.e., surrogate inputs that are optimised upon training to avoid building (kernel) interactions between each pair of observations in the dataset. In short, we propose a method that adapts kernels to the set up of high-dimensional and potentially sparse datasets. To illustrate our approach, we test it on four well-known datasets and benchmark its results against most available models. While comparisons are difficult and should be interpreted carefully, KFM is able to perform well and obtains the best performance overall. Our methodology is not limited to recommender systems and can be applied to other settings, which we illustrate on a heart disease classification task.