Optimal Rates of Approximation by Shallow ReLU $$^k$$ Neural Networks and Applications to Nonparametric Regression

IF 4.6 Q2 MATERIALS SCIENCE, BIOMATERIALS ACS Applied Bio Materials Pub Date : 2024-02-26 DOI:10.1007/s00365-024-09679-z

{"title":"Optimal Rates of Approximation by Shallow ReLU $$^k$$ Neural Networks and Applications to Nonparametric Regression","authors":"","doi":"10.1007/s00365-024-09679-z","DOIUrl":null,"url":null,"abstract":"<h3>Abstract</h3> We study the approximation capacity of some variation spaces corresponding to shallow ReLU \$^k\$ neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU \$^k\$ neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.","PeriodicalId":2,"journal":{"name":"ACS Applied Bio Materials","volume":null,"pages":null},"PeriodicalIF":4.6000,"publicationDate":"2024-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"ACS Applied Bio Materials","FirstCategoryId":"100","ListUrlMain":"https://doi.org/10.1007/s00365-024-09679-z","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MATERIALS SCIENCE, BIOMATERIALS","Score":null,"Total":0}

引用次数: 0

Abstract

We study the approximation capacity of some variation spaces corresponding to shallow ReLU $^k$ neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU $^k$ neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

浅 ReLU $$^k$$ 神经网络的最佳逼近率及其在非参数回归中的应用

摘要我们研究了与浅层 ReLU （^k\ ）神经网络相对应的一些变化空间的逼近能力。研究表明，足够光滑的函数都包含在这些具有有限变化规范的空间中。对于光滑度较低的函数，我们建立了以变化规范为单位的逼近率。利用这些结果，我们能够证明浅层 ReLU $^k$神经网络在神经元数量方面的最优逼近率。我们还展示了这些结果如何用于推导深度神经网络和卷积神经网络（CNN）的近似边界。作为应用，我们使用三种 ReLU 神经网络模型研究了非参数回归的收敛率：浅层神经网络、过参数化神经网络和 CNN。特别是，我们证明了浅层神经网络在学习赫尔德函数时可以达到最小最优率，这是对深度神经网络最新成果的补充。此外，我们还证明了过参数化（深层或浅层）神经网络在非参数回归方面可以达到近乎最优的速率。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊