{"title":"Deep Linear Networks for Matrix Completion—an Infinite Depth Limit","authors":"Nadav Cohen, Govind Menon, Zsolt Veraszto","doi":"10.1137/22m1530653","DOIUrl":null,"url":null,"abstract":"SIAM Journal on Applied Dynamical Systems, Volume 22, Issue 4, Page 3208-3232, December 2023. <br/> Abstract.The deep linear network (DLN) is a model for implicit regularization in gradient based optimization of overparametrized learning architectures. Training the DLN corresponds to a Riemannian gradient flow, where the Riemannian metric is defined by the architecture of the network and the loss function is defined by the learning task. We extend this geometric framework, obtaining explicit expressions for the volume form, including the case when the network has infinite depth. We investigate the link between the Riemannian geometry and the training asymptotics for matrix completion with rigorous analysis and numerics. We propose that under small initialization, implicit regularization is a result of bias towards high state space volume.","PeriodicalId":1,"journal":{"name":"Accounts of Chemical Research","volume":null,"pages":null},"PeriodicalIF":16.4000,"publicationDate":"2023-11-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Accounts of Chemical Research","FirstCategoryId":"100","ListUrlMain":"https://doi.org/10.1137/22m1530653","RegionNum":1,"RegionCategory":"化学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"CHEMISTRY, MULTIDISCIPLINARY","Score":null,"Total":0}
引用次数: 0
Abstract
SIAM Journal on Applied Dynamical Systems, Volume 22, Issue 4, Page 3208-3232, December 2023. Abstract.The deep linear network (DLN) is a model for implicit regularization in gradient based optimization of overparametrized learning architectures. Training the DLN corresponds to a Riemannian gradient flow, where the Riemannian metric is defined by the architecture of the network and the loss function is defined by the learning task. We extend this geometric framework, obtaining explicit expressions for the volume form, including the case when the network has infinite depth. We investigate the link between the Riemannian geometry and the training asymptotics for matrix completion with rigorous analysis and numerics. We propose that under small initialization, implicit regularization is a result of bias towards high state space volume.
应用动力系统学报，第22卷，第4期，第3208-3232页，2023年12月。摘要。深度线性网络(deep linear network, DLN)是一种基于梯度优化的隐式正则化模型。训练DLN对应于黎曼梯度流，其中黎曼度量由网络的体系结构定义，损失函数由学习任务定义。我们扩展了这个几何框架，得到了体积形式的显式表达式，包括网络具有无限深度的情况。我们用严格的分析和数值研究了黎曼几何和矩阵补全的训练渐近之间的联系。我们提出在小初始化下，隐式正则化是偏向于高状态空间体积的结果。
期刊介绍：
Accounts of Chemical Research presents short, concise and critical articles offering easy-to-read overviews of basic research and applications in all areas of chemistry and biochemistry. These short reviews focus on research from the author’s own laboratory and are designed to teach the reader about a research project. In addition, Accounts of Chemical Research publishes commentaries that give an informed opinion on a current research problem. Special Issues online are devoted to a single topic of unusual activity and significance.
Accounts of Chemical Research replaces the traditional article abstract with an article "Conspectus." These entries synopsize the research affording the reader a closer look at the content and significance of an article. Through this provision of a more detailed description of the article contents, the Conspectus enhances the article's discoverability by search engines and the exposure for the research.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
