Parametrization of Statistical Models in Three-layer Neural Networks

Abstract

In a hierarchical structure model such as a neural network, the set of true parameters consists of not one point but a union of several manifolds and contains complicated singularities, making it difficult to analyze their behavior and discuss it theoretically. We first consider that the set of true parameters that is realizable by a statistical model with a hyperbolic tangent as an activation function are algebraic sets defined by finite polynomials. The main purpose of this paper is to show the parametrization of algebraic sets containing complicated singularities in three-layer neural networks using the Cröbner basis technique for finitely generated ideals.