A self-organized MoE framework for distributed federated learning

Abstract

Federated Learning (FL) has solved the problem of data silos by enabling multiple participants to cooperatively train a global model while ensuring data privacy; however, it is still a challenge to establish a Distributed Federated Learning (DFL) framework that naturally suffers from the heterogeneity of devices and datasets. Rather than conventional FL algorithms that combine client models for a single global model, a Mixture of Experts (MoE) based FL is an effective alternative that can admit individual features on each client dataset by partitioning the entire latent space. In this study, we introduce the Self-Organized MoE Framework (SOMFed), which enhances the DFL lifecycle under asynchronous updates and statistical challenges of datasets. Considering that nodes are assumed to lack label information in contrast to most of previous studies, aside from their class data, we propose the Model Assessment and Selection (MASS) algorithm for the SOMFed framework, leveraging self-supervised learning. It evaluates and chooses suitable experts for own unlabeled dataset by differentiating the performance of the representation layers among experts using Bayesian optimization and Conditional Loss Adjustment (CLA). The SOMFed exhibits superior performance in extensive experiments with different non-IID distributions and stragglers compared to FedAVG, FedAsync, SCAFFOLD, FedAT, and Adaptive Expert Models (AEM). In particular, it demonstrates robustness against pathological non-IID distribution on CIFAR10, achieving accuracy of 79.42%.