Applying Robust Gradient Difference Compression to Federated Learning

Abstract

Nowadays, federated learning has been a prevailing paradigm for large-scale distributed machine learning, which is faced with the problem of communication bottleneck. To solve this problem, recent works usually apply different compression techniques such as sparsification and quantization compressors. However, such approaches are all lossy compression and have two drawbacks. First, they could lead to information loss of the global parameter. Second, compressed parameters carrying less information would be more likely to be attacked by malicious workers than full parameters, leading to a Byzantine failure of the model. In this paper, to avoid information loss, mitigate the communication bottleneck, and at the same time tolerate popular Byzantine attacks, we propose FedGraD, which leverages gradient difference compression and combines robust aggregation rules in federated learning settings. Our experimental results on three different datasets a9a, w8a and mushrooms show good performance of our method.