Quantized Compressed Sensing for Communication-Efficient Federated Learning

Abstract

Federated learning (FL) is a decentralized artificial intelligence technique for training a global model on a parameter server (PS) through collaboration with wireless devices, each with its own local training data set. In this paper, we present a communication-efficient FL framework which consists of gradient compression and reconstruction strategies based on quantized compressed sensing (QCS). The key idea of the gradient compression strategy is to compress-and-quantize a local gradient vector computed at each device after sparsifying this vector in a block wise fashion. Our gradient compression strategy can make communication overhead less than one bit per gradient entry. For accurate reconstruction of the local gradient from the compressed signals at the PS, we employ a expectation-maximization generalized-approximate-message-passing algorithm. The algorithm iteratively computes an approximate minimum mean square error solution of the local gradient, while learning the unknown model parameters of the Bernoulli Gaussian-mixture prior. Using the MNIST data set, we demonstrate that the presented FL framework can achieve almost identical classification performance with the case that performs no compression, while achieving a significant reduction of communication overhead.