Efficient and Structural Gradient Compression with Principal Component Analysis for Distributed Training

Abstract

Distributed machine learning is a promising machine learning approach for academia and industry. It can generate a machine learning model for dispersed training data via iterative training in a distributed fashion. To speed up the training process of distributed machine learning, it is essential to reduce the communication load among training nodes. In this paper, we propose a layer-wise gradient compression scheme based on principal component analysis and error accumulation. The key of our solution is to consider the gradient characteristics and architecture of neural networks by taking advantage of the compression ability enabled by PCA and the feedback ability enabled by error accumulation. The preliminary results on image classification task show that our scheme achieves good performance and reduces 97% of the gradient transmission.