On the Convergence of Decentralized Stochastic Gradient Descent With Biased Gradients

Abstract

Stochastic optimization algorithms are widely used to solve large-scale machine learning problems. However, their theoretical analysis necessitates access to unbiased estimates of the true gradients. To address this issue, we perform a comprehensive convergence rate analysis of stochastic gradient descent (SGD) with biased gradients for decentralized optimization. In non-convex settings, we show that for decentralized SGD utilizing biased gradients, the gradient in expectation is bounded asymptotically at a rate of $\mathcal{O}(1/\sqrt{nT}+n/T)$, and the bound is linearly correlated to the biased gradient gap. In particular, we can recover the convergence results in the unbiased stochastic gradient setting when the biased gradient gap is zero. Lastly, we provide empirical support for our theoretical findings through extensive numerical experiments.