Snowball Effect in Federated Learning: An Approach of Exponentially Expanding Structures for Optimizing the Training Efficiency

Abstract

Efficient federated learning (FL) in mobile edge networks faces challenges due to energy-consuming on-device training and wireless transmission. Optimizing the neural network structures is an effective approach to achieving energy savings. In this paper, we present a Snowball FL training with expanding neural network structure, which starts with a small-sized submodel and gradually progresses to a full-sized model. To achieve this, we first design the submodel and embedding extraction schemes for fine-grained model structure expansion. We then investigate the joint minimization problem of the global training loss and system-wise energy consumption. After that, we decompose the optimization problem into a long-term model structure expansion subproblem and a single-round local resource allocation subproblem. Specifically, the former subproblem is transformed into a variational calculus problem by leveraging theoretical analysis of the convergence bound. The Euler-Lagrange method is used to derive the solution, where the optimal evolution strategy for the model structure exponentially increases with the global round (i.e., the Snowball effect). Meanwhile, the latter subproblem is solved by convex optimization to acquire the optimal computing frequency and transmission power. Experiments indicate that the proposed framework can save about 50% of energy consumption to achieve on-par accuracy against state-of-the-art algorithms.