Exponential expressivity of ReLUk neural networks on Gevrey classes with point singularities

Abstract

We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains D ⊂ ℝ^d, d = 2, 3. We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in D, comprising the countably-normed spaces of I. M. Babuska and B. Q. Guo.

As intermediate result, we prove that continuous, piecewise polynomial high order (“p-version”) finite elements with elementwise polynomial degree p ∈ ℕ on arbitrary, regular, simplicial partitions of polyhedral domains D ⊂ ℝ^d, d ⩾ 2, can be exactly emulated by neural networks combining ReLU and ReLU² activations.

On shape-regular, simplicial partitions of polytopal domains D, both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the hp finite element space of I. M. Babuška and B. Q. Guo.