{"title":"神经 ODE 中插值的深度和宽度之间的相互作用","authors":"","doi":"10.1016/j.neunet.2024.106640","DOIUrl":null,"url":null,"abstract":"<div><p>Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width <span><math><mi>p</mi></math></span> and the number of transitions between layers <span><math><mi>L</mi></math></span> (corresponding to a depth of <span><math><mrow><mi>L</mi><mo>+</mo><mn>1</mn></mrow></math></span>). Specifically, we construct explicit controls interpolating either a finite dataset <span><math><mi>D</mi></math></span>, comprising <span><math><mi>N</mi></math></span> pairs of points in <span><math><msup><mrow><mi>R</mi></mrow><mrow><mi>d</mi></mrow></msup></math></span>, or two probability measures within a Wasserstein error margin <span><math><mrow><mi>ɛ</mi><mo>></mo><mn>0</mn></mrow></math></span>. Our findings reveal a balancing trade-off between <span><math><mi>p</mi></math></span> and <span><math><mi>L</mi></math></span>, with <span><math><mi>L</mi></math></span> scaling as <span><math><mrow><mn>1</mn><mo>+</mo><mi>O</mi><mrow><mo>(</mo><mi>N</mi><mo>/</mo><mi>p</mi><mo>)</mo></mrow></mrow></math></span> for data interpolation, and as <span><math><mrow><mn>1</mn><mo>+</mo><mi>O</mi><mfenced><mrow><msup><mrow><mi>p</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo>+</mo><msup><mrow><mrow><mo>(</mo><mn>1</mn><mo>+</mo><mi>p</mi><mo>)</mo></mrow></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><msup><mrow><mi>ɛ</mi></mrow><mrow><mo>−</mo><mi>d</mi></mrow></msup></mrow></mfenced></mrow></math></span> for measures.</p><p>In the high-dimensional and wide setting where <span><math><mrow><mi>d</mi><mo>,</mo><mi>p</mi><mo>></mo><mi>N</mi></mrow></math></span>, our result can be refined to achieve <span><math><mrow><mi>L</mi><mo>=</mo><mn>0</mn></mrow></math></span>. This naturally raises the problem of data interpolation in the autonomous regime, characterized by <span><math><mrow><mi>L</mi><mo>=</mo><mn>0</mn></mrow></math></span>. We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when <span><math><mrow><mi>p</mi><mo>=</mo><mi>N</mi></mrow></math></span> we develop an inductive control strategy based on a separability assumption whose probability increases with <span><math><mi>d</mi></math></span>. In the second one, we establish an explicit error decay rate with respect to <span><math><mi>p</mi></math></span> which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating <span><math><mi>D</mi></math></span>.</p></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":null,"pages":null},"PeriodicalIF":6.0000,"publicationDate":"2024-08-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0893608024005641/pdfft?md5=480cae19d4a2c169ff78cc6025a33eae&pid=1-s2.0-S0893608024005641-main.pdf","citationCount":"0","resultStr":"{\"title\":\"Interplay between depth and width for interpolation in neural ODEs\",\"authors\":\"\",\"doi\":\"10.1016/j.neunet.2024.106640\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width <span><math><mi>p</mi></math></span> and the number of transitions between layers <span><math><mi>L</mi></math></span> (corresponding to a depth of <span><math><mrow><mi>L</mi><mo>+</mo><mn>1</mn></mrow></math></span>). Specifically, we construct explicit controls interpolating either a finite dataset <span><math><mi>D</mi></math></span>, comprising <span><math><mi>N</mi></math></span> pairs of points in <span><math><msup><mrow><mi>R</mi></mrow><mrow><mi>d</mi></mrow></msup></math></span>, or two probability measures within a Wasserstein error margin <span><math><mrow><mi>ɛ</mi><mo>></mo><mn>0</mn></mrow></math></span>. Our findings reveal a balancing trade-off between <span><math><mi>p</mi></math></span> and <span><math><mi>L</mi></math></span>, with <span><math><mi>L</mi></math></span> scaling as <span><math><mrow><mn>1</mn><mo>+</mo><mi>O</mi><mrow><mo>(</mo><mi>N</mi><mo>/</mo><mi>p</mi><mo>)</mo></mrow></mrow></math></span> for data interpolation, and as <span><math><mrow><mn>1</mn><mo>+</mo><mi>O</mi><mfenced><mrow><msup><mrow><mi>p</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo>+</mo><msup><mrow><mrow><mo>(</mo><mn>1</mn><mo>+</mo><mi>p</mi><mo>)</mo></mrow></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><msup><mrow><mi>ɛ</mi></mrow><mrow><mo>−</mo><mi>d</mi></mrow></msup></mrow></mfenced></mrow></math></span> for measures.</p><p>In the high-dimensional and wide setting where <span><math><mrow><mi>d</mi><mo>,</mo><mi>p</mi><mo>></mo><mi>N</mi></mrow></math></span>, our result can be refined to achieve <span><math><mrow><mi>L</mi><mo>=</mo><mn>0</mn></mrow></math></span>. This naturally raises the problem of data interpolation in the autonomous regime, characterized by <span><math><mrow><mi>L</mi><mo>=</mo><mn>0</mn></mrow></math></span>. We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when <span><math><mrow><mi>p</mi><mo>=</mo><mi>N</mi></mrow></math></span> we develop an inductive control strategy based on a separability assumption whose probability increases with <span><math><mi>d</mi></math></span>. In the second one, we establish an explicit error decay rate with respect to <span><math><mi>p</mi></math></span> which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating <span><math><mi>D</mi></math></span>.</p></div>\",\"PeriodicalId\":49763,\"journal\":{\"name\":\"Neural Networks\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":6.0000,\"publicationDate\":\"2024-08-19\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.sciencedirect.com/science/article/pii/S0893608024005641/pdfft?md5=480cae19d4a2c169ff78cc6025a33eae&pid=1-s2.0-S0893608024005641-main.pdf\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Neural Networks\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S0893608024005641\",\"RegionNum\":1,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Neural Networks","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0893608024005641","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
摘要
从控制的角度来看,神经常微分方程已成为监督学习的一种天然工具,但人们对其结构所起作用的全面了解却仍然遥不可及。在这项工作中,我们研究了宽度 p 与层间转换次数 L(对应深度 L+1)之间的相互作用。具体来说,我们构建了明确的控制方法,既可以对由 Rd 中 N 对点组成的有限数据集 D 进行插值,也可以对 Wasserstein 误差范围ɛ>0 内的两个概率度量进行插值。我们的发现揭示了 p 和 L 之间的平衡权衡,对于数据插值,L 的缩放为 1+O(N/p),而对于度量,L 的缩放为 1+Op-1+(1+p)-1ɛ-d。在 d,p>N 的高维和宽范围设置中,我们的结果可以细化到 L=0。在第一种情况下,当 p=N 时,我们基于可分性假设开发了一种归纳控制策略,其概率随 d 的增加而增加。在第二种情况下,我们建立了一个与 p 有关的显式误差衰减率,该误差衰减率是将通用近似定理应用于定制的利普斯奇茨矢量场插值 D 的结果。
Interplay between depth and width for interpolation in neural ODEs
Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width and the number of transitions between layers (corresponding to a depth of ). Specifically, we construct explicit controls interpolating either a finite dataset , comprising pairs of points in , or two probability measures within a Wasserstein error margin . Our findings reveal a balancing trade-off between and , with scaling as for data interpolation, and as for measures.
In the high-dimensional and wide setting where , our result can be refined to achieve . This naturally raises the problem of data interpolation in the autonomous regime, characterized by . We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when we develop an inductive control strategy based on a separability assumption whose probability increases with . In the second one, we establish an explicit error decay rate with respect to which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating .
期刊介绍:
Neural Networks is a platform that aims to foster an international community of scholars and practitioners interested in neural networks, deep learning, and other approaches to artificial intelligence and machine learning. Our journal invites submissions covering various aspects of neural networks research, from computational neuroscience and cognitive modeling to mathematical analyses and engineering applications. By providing a forum for interdisciplinary discussions between biology and technology, we aim to encourage the development of biologically-inspired artificial intelligence.