DIN: A Decentralized Inexact Newton Algorithm for Consensus Optimization

Abstract

This paper tackles a challenging decentralized consensus optimization problem defined over a network of interconnected devices. The devices work collaboratively to solve a problem using only their local data and exchanging information with their immediate neighbors. One approach to solving such a problem is to use Newton-type methods, which are known for their fast convergence. However, these methods have a significant drawback as they require transmitting Hessian information between devices. This not only makes them communication-inefficient but also raises privacy concerns. To address these issues, we present a novel approach that transforms the Newton direction learning problem into a formulation composed of a sum of separable functions subjected to a consensus constraint and learns an inexact Newton direction alongside the global model without enforcing devices to share their computed Hessians using the proximal primal-dual (Prox-PDA) algorithm. Our algorithm, coined DIN, avoids sharing Hessian information between devices since each device shares a model-sized vector, concealing the first- and second-order information, reducing the network’s burden and improving both communication and energy efficiencies. Furthermore, we prove that DIN descent direction converges linearly to the optimal Newton direction. Numerical simulations corroborate that DIN exhibits higher communication efficiency in terms of communication rounds while consuming less communication and computation energy compared to existing second-order decentralized baselines.