Alleviating straggler impacts for data parallel deep learning with hybrid parameter update

Abstract

Data parallelism in distributed clusters faces challenges due to costly global parameter updates and performance imbalances, leading to stragglers that negatively impact training speed and accuracy. This paper proposes Cooperate Grouping Parallel (CGP), a hybrid parameter update scheme to alleviate the problem. CGP supports dynamic grouping among parallel workers and utilizes both intra-group synchronous update and inter-group asynchronous update. CGP treats straggler as an opportunity for worker groups to cooperatively reduce the global parameter update cost. We give the theoretical upper bound of model accuracy deviation caused by inter-group asynchronous updates, which proves the convergence property of the proposed CGP. Extensive testbed experiments on different workloads shows that CGP achieves 1.94$\times$ speedup compared to the other methods on average in different scenarios, and CGP improves accuracy by 16.8% over the asynchronous methods.