Decentralized learning over a network with Nyström approximation using SGD

Abstract

Nowadays we often meet with a learning problem when data are distributed on different machines connected via a network, instead of stored centrally. Here we consider decentralized supervised learning in a reproducing kernel Hilbert space. We note that standard gradient descent in a reproducing kernel Hilbert space is difficult to implement with multiple communications between worker machines. On the other hand, the Nyström approximation using gradient descent is more suited for the decentralized setting since only a small number of data points need to be shared at the beginning of the algorithm. In the setting of decentralized distributed learning in a reproducing kernel Hilbert space, we establish the optimal learning rate of stochastic gradient descent based on mini-batches, allowing multiple passes over the data set. The proposal provides a scalable approach to nonparametric estimation combining gradient method, distributed estimation, and random projection.