Exploring Weight Distributions and Dependence in Neural Networks With $\alpha$-Stable Distributions

Abstract

The fundamental use of neural networks is in providing a nonlinear mapping between input and output data with possibly a high number of parameters that can be learned from data directly. Consequently, studying the model's parameters, particularly the weights, is of paramount importance. The distribution and interdependencies of these weights have a direct impact on the model's generalizability, compressibility, initialization, and convergence speed. By fitting the weights of pretrained neural networks using the $\alpha$ -stable distributions and conducting statistical tests, we discover widespread heavy-tailed phenomena in neural network weights, with a few layers exhibiting asymmetry. Additionally, we employ a multivariate $\alpha$ -stable distribution to model the weights and explore the relationship between weights within and across layers by calculating the signed symmetric covariation coefficient. The results reveal a strong dependence among certain weights. Our findings indicate that the Gaussian assumption, symmetry assumption, and independence assumption commonly used in neural network research might be inconsistent with reality. In conclusion, our research shows three properties observed in neural network weights: heavy-tailed phenomena, asymmetry, and dependence on certain weights.