Adapter-guided knowledge transfer for heterogeneous federated learning

Abstract

Federated learning (FL) aims to collaboratively train a global model or multiple local models on decentralized data. Most existing FL approaches focus on addressing statistical heterogeneity among clients, often overlooking the challenge of model heterogeneity. To address both statistical and model heterogeneity issues, we propose FedAKT, a novel model-heterogeneous personalized federated learning (MHPFL) approach. First, to facilitate cross-client knowledge transfer, our method adds a small homogeneous adapter for each client. Second, we introduce a feature-based mutual distillation (FMD) mechanism, which promotes bidirectional knowledge exchange in local models. Third, a header dual-use (HDU) mechanism is proposed, enabling each local model’s header to effectively learn feature information from different perspectives. Extensive experiments on the CIFAR10, CIFAR-100, and Tiny-ImageNet datasets demonstrate the superiority of FedAKT compared to advanced baselines.