Differentially private federated learning with non-IID data

Abstract

In Differentially Private Federated Learning (DPFL), gradient clipping and random noise addition disproportionately affect statistically heterogeneous data. As a consequence, DPFL has a disparate impact: the accuracy of models trained with DPFL tends to decrease more on these data. If the accuracy of the original model decreases on heterogeneous data, DPFL may degrade the accuracy performance more. In this work, we study the utility loss inequality due to differential privacy and compare the convergence of the private and non-private models. Specifically, we analyze the gradient differences caused by statistically heterogeneous data and explain how statistical heterogeneity relates to the effect of privacy on model convergence. In addition, we propose an improved DPFL algorithm, called R-DPFL, to achieve differential privacy at the same cost but with good utility. R-DPFL adjusts the gradient clipping value and the number of selected users at beginning according to the degree of statistical heterogeneity of the data, and weakens the direct proportional relationship between the differential privacy and the gradient difference, thereby reducing the impact of differential privacy on the model trained on heterogeneous data. Our experimental evaluation shows the effectiveness of our elimination algorithm in achieving the same cost of differential privacy with satisfactory utility. Our code is publicly available at https://github.com/chengshuyan/R-DPFL.