Balancing Rates and Variance via Adaptive Batch-Sizes in First-Order Stochastic Optimization

Zhan Gao, Alec Koppel, Alejandro Ribeiro
{"title":"Balancing Rates and Variance via Adaptive Batch-Sizes in First-Order Stochastic Optimization","authors":"Zhan Gao, Alec Koppel, Alejandro Ribeiro","doi":"10.1109/ICASSP40776.2020.9054292","DOIUrl":null,"url":null,"abstract":"Stochastic gradient descent is a canonical tool for addressing stochastic optimization problems, and forms the bedrock of modern machine learning and statistics. In this work, we seek to balance the fact that attenuating step-sizes is required for exact asymptotic convergence with the fact that larger constant step-sizes learn faster in finite time up to an error. To do so, rather than fixing the mini-batch and step-size at the outset, we propose a strategy to allow parameters to evolve adaptively. Specifically, the batch-size is set to be a piecewise-constant increasing sequence where the increase occurs when a suitable error criterion is satisfied. Moreover, the step-size is selected as that which yields the fastest convergence. The overall algorithm, two scale adaptive (TSA) scheme, is shown to inherit the exact asymptotic convergence of stochastic gradient method. More importantly, the optimal error decreasing rate is achieved theoretically, as well as an overall reduction in sample computational cost. Experimentally, we observe a favorable tradeoff relative to standard SGD schemes absorbing their advantages, which illustrates the significant performance of proposed TSA scheme.","PeriodicalId":13127,"journal":{"name":"ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","volume":"45 1","pages":"5385-5389"},"PeriodicalIF":0.0000,"publicationDate":"2020-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICASSP40776.2020.9054292","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1

Abstract

Stochastic gradient descent is a canonical tool for addressing stochastic optimization problems, and forms the bedrock of modern machine learning and statistics. In this work, we seek to balance the fact that attenuating step-sizes is required for exact asymptotic convergence with the fact that larger constant step-sizes learn faster in finite time up to an error. To do so, rather than fixing the mini-batch and step-size at the outset, we propose a strategy to allow parameters to evolve adaptively. Specifically, the batch-size is set to be a piecewise-constant increasing sequence where the increase occurs when a suitable error criterion is satisfied. Moreover, the step-size is selected as that which yields the fastest convergence. The overall algorithm, two scale adaptive (TSA) scheme, is shown to inherit the exact asymptotic convergence of stochastic gradient method. More importantly, the optimal error decreasing rate is achieved theoretically, as well as an overall reduction in sample computational cost. Experimentally, we observe a favorable tradeoff relative to standard SGD schemes absorbing their advantages, which illustrates the significant performance of proposed TSA scheme.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
一阶随机优化中自适应批大小的平衡率和方差
随机梯度下降是解决随机优化问题的规范工具,是现代机器学习和统计学的基础。在这项工作中,我们试图平衡这样一个事实,即衰减步长是精确渐近收敛所必需的,而更大的恒定步长在有限时间内学习得更快,直到一个误差。为了做到这一点,我们提出了一种允许参数自适应进化的策略,而不是在一开始就固定小批量和步长。具体来说,将批大小设置为分段常数递增序列,当满足适当的误差标准时,增加就会发生。此外,步长选择为产生最快的收敛。总体算法双尺度自适应(TSA)方案继承了随机梯度法的精确渐近收敛性。更重要的是,从理论上实现了最优的误码率,以及总体上减少了样本计算成本。实验中,我们观察到相对于标准SGD方案的有利权衡吸收了它们的优点,这说明了所提出的TSA方案的显著性能。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
Theoretical Analysis of Multi-Carrier Agile Phased Array Radar Paco and Paco-Dct: Patch Consensus and Its Application To Inpainting Array-Geometry-Aware Spatial Active Noise Control Based on Direction-of-Arrival Weighting Neural Network Wiretap Code Design for Multi-Mode Fiber Optical Channels Distributed Non-Orthogonal Pilot Design for Multi-Cell Massive Mimo Systems
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1