GWPF: Communication-efficient federated learning with Gradient-Wise Parameter Freezing

IF 4.4 2区 计算机科学 Q1 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE Computer Networks Pub Date : 2024-11-04 DOI:10.1016/j.comnet.2024.110886
Duo Yang , Yunqi Gao , Bing Hu , A-Long Jin , Wei Wang , Yang You
{"title":"GWPF: Communication-efficient federated learning with Gradient-Wise Parameter Freezing","authors":"Duo Yang ,&nbsp;Yunqi Gao ,&nbsp;Bing Hu ,&nbsp;A-Long Jin ,&nbsp;Wei Wang ,&nbsp;Yang You","doi":"10.1016/j.comnet.2024.110886","DOIUrl":null,"url":null,"abstract":"<div><div>Communication bottleneck is a critical challenge in federated learning. While parameter freezing has emerged as a popular approach, utilizing fine-grained parameters as aggregation objects, existing methods suffer from issues such as a lack of thawing strategy, lag and inflexibility in the thawing process, and underutilization of frozen parameters’ updates. To address these challenges, we propose Gradient-Wise Parameter Freezing (GWPF), a mechanism that wisely controls frozen periods for different parameters through parameter freezing and thawing strategies. GWPF globally freezes parameters with insignificant gradients and excludes frozen parameters from global updates during the frozen period, reducing communication overhead and accelerating training. The thawing strategy, based on global decisions by the server and collaboration with clients, leverages real-time feedback on the locally accumulated gradients of frozen parameters in each round, achieving a balanced approach between mitigating communication and enhancing model accuracy. We provide theoretical analysis and a convergence guarantee for non-convex objectives. Extensive experiments confirm that our mechanism achieves a speedup of up to 4.52 times in time-to-accuracy performance and reduces communication overhead by up to 48.73%. It also improves final model accuracy by up to 2.01% compared to the existing fastest method APF. The code for GWPF is available at <span><span>https://github.com/Dora233/GWPF</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":50637,"journal":{"name":"Computer Networks","volume":"255 ","pages":"Article 110886"},"PeriodicalIF":4.4000,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Computer Networks","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1389128624007187","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, HARDWARE & ARCHITECTURE","Score":null,"Total":0}
引用次数: 0

Abstract

Communication bottleneck is a critical challenge in federated learning. While parameter freezing has emerged as a popular approach, utilizing fine-grained parameters as aggregation objects, existing methods suffer from issues such as a lack of thawing strategy, lag and inflexibility in the thawing process, and underutilization of frozen parameters’ updates. To address these challenges, we propose Gradient-Wise Parameter Freezing (GWPF), a mechanism that wisely controls frozen periods for different parameters through parameter freezing and thawing strategies. GWPF globally freezes parameters with insignificant gradients and excludes frozen parameters from global updates during the frozen period, reducing communication overhead and accelerating training. The thawing strategy, based on global decisions by the server and collaboration with clients, leverages real-time feedback on the locally accumulated gradients of frozen parameters in each round, achieving a balanced approach between mitigating communication and enhancing model accuracy. We provide theoretical analysis and a convergence guarantee for non-convex objectives. Extensive experiments confirm that our mechanism achieves a speedup of up to 4.52 times in time-to-accuracy performance and reduces communication overhead by up to 48.73%. It also improves final model accuracy by up to 2.01% compared to the existing fastest method APF. The code for GWPF is available at https://github.com/Dora233/GWPF.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
GWPF:利用渐进参数冻结技术进行通信效率高的联合学习
通信瓶颈是联合学习中的一个关键挑战。虽然利用细粒度参数作为聚合对象的参数冻结已成为一种流行的方法,但现有方法存在一些问题,如缺乏解冻策略、解冻过程滞后且缺乏灵活性,以及未充分利用冻结参数的更新。为了应对这些挑战,我们提出了渐进-明智参数冻结(GWPF),这是一种通过参数冻结和解冻策略明智控制不同参数冻结期的机制。GWPF 全局冻结梯度不显著的参数,并在冻结期间将冻结参数排除在全局更新之外,从而减少通信开销并加速训练。解冻策略基于服务器的全局决策和与客户端的协作,利用每轮冻结参数的本地累积梯度的实时反馈,在减少通信和提高模型精度之间实现平衡。我们为非凸目标提供了理论分析和收敛保证。广泛的实验证实,我们的机制在时间-精度性能方面最多可加快 4.52 倍,通信开销最多可减少 48.73%。与现有的最快方法 APF 相比,它还将最终模型精度提高了 2.01%。GWPF 的代码见 https://github.com/Dora233/GWPF。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Computer Networks
Computer Networks 工程技术-电信学
CiteScore
10.80
自引率
3.60%
发文量
434
审稿时长
8.6 months
期刊介绍: Computer Networks is an international, archival journal providing a publication vehicle for complete coverage of all topics of interest to those involved in the computer communications networking area. The audience includes researchers, managers and operators of networks as well as designers and implementors. The Editorial Board will consider any material for publication that is of interest to those groups.
期刊最新文献
Performance modeling and comparison of URLLC and eMBB coexistence strategies in 5G new radio systems Integrating Unmanned Aerial Vehicles (UAVs) with Vehicular Ad-hoc NETworks (VANETs): Architectures, applications, opportunities Deep reinforcement learning for autonomous SideLink radio resource management in platoon-based C-V2X networks: An overview Robust and energy-efficient RPL optimization algorithm with scalable deep reinforcement learning for IIoT Privacy-preserving local clustering coefficient query on structured encrypted graphs
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1