Communication-Efficient and Privacy-Aware Distributed Learning

Abstract

Communication efficiency and privacy are two key concerns in modern distributed computing systems. Towards this goal, this article proposes partial sharing private distributed learning (PPDL) algorithms that offer communication efficiency while preserving privacy, thus making them suitable for applications with limited resources in adversarial environments. First, we propose a noise injection-based PPDL algorithm that achieves communication efficiency by sharing only a fraction of the information at each consensus iteration and provides privacy by perturbing the information exchanged among neighbors. To further increase privacy, local information is randomly decomposed into private and public substates before sharing with the neighbors. This results in a decomposition- and noise-injection-based PPDL strategy in which only a freaction of the perturbeesd public substate is shared during local collaborations, whereas the private substate is updated locally without being shared. To determine the impact of communication savings and privacy preservation on the performance of distributed learning algorithms, we analyze the mean and mean-square convergence of the proposed algorithms. Moreover, we investigate the privacy of agents by characterizing privacy as the mean squared error of the estimate of private information at the honest-but-curious adversary. The analytical results show a tradeoff between communication efficiency and privacy in proposed PPDL algorithms, while decomposition- and noise-injection-based PPDL improves privacy compared to noise-injection-based PPDL. Lastly, numerical simulations corroborate the analytical findings.