Clustered Federated Multi-Task Learning with Non-IID Data

Abstract

Federated Learning enables the collaborative learning in cross-client scenarios while keeping the clients' data local for privacy. The presence of non-IID data is one of major challenges in federated learning. To deal with this statistic challenge, federated multi-task learning considers the local training for each client as a single task. However, all the clients must participate in each training round, and it is inapplicable to mobile or IOT devices with constrained communication capability. To achieve the communication-efficiency and high accuracy with non-IID data, we propose a clustered federated multi-task learning by exploring client clustering and multi-task learning. We measure the similarities of local data among clients indirectly through their models' parameters, and design a client clustering strategy to enable clients with similar data distribution into a same group. The limitation of full-participation can be eliminated through the way of model training for groups instead of individual clients. The convergence analysis and experimental evaluation on real-world datasets shows that our work outperforms the basic federated learning in accuracy and is also more communication-efficient than the existing federated multi-task learning.