A novel staged training strategy leveraging knowledge distillation and model fusion for heterogeneous federated learning

Abstract

Client-side data heterogeneity poses a significant challenge in Federated Learning (FL), limiting the effectiveness of global models. To address this, we propose a staged training approach combining Knowledge Distillation and model fusion. First, a regularized KD technique trains a robust teacher model on the server, transferring knowledge to student models to enhance convergence and reduce overfitting. Then, an adaptive parameter assignment mechanism intelligently combines the local and global models, enabling clients to integrate global knowledge with local features for improved accuracy. Experimental results on multiple image classification datasets demonstrate that our approach outperforms existing algorithms in both convergence speed and accuracy, particularly in highly heterogeneous scenarios. It effectively balances the global model’s generalization and local personalization, providing a robust solution for FL.