Spectral complexity of deep neural networks

2024-05-15

arXiv.org

AI Summary

This paper addresses the complexity characterization of fully connected deep neural networks, aiming to elucidate how depth growth and activation function choice (e.g., ReLU) affect generalization and structural properties. We propose a complexity analysis framework based on the angular power spectrum of the induced Gaussian process and establish, under the infinite-depth limit with fixed width-to-depth ratio, the asymptotic distribution theory of the spectral sequence. We introduce, for the first time, three spectral-structure-based complexity criteria—low-disorder, sparse, and high-disorder—rigorously uncovering activation-specific properties such as the intrinsic sparsity of ReLU networks. Integrating weak convergence of stochastic processes, spherical harmonic analysis, and asymptotic statistics, we derive verifiable spectral characterizations and classification criteria. All theoretical findings are systematically validated through comprehensive numerical experiments.

Technology Category

• Machine Learning: Deep Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Social Networks and Social Media: Social media analysis through the lenses of networks

Abstract

It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting field to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

Problem

• Deep Learning
• Neural Network Complexity
• Activation Functions

Innovation

• Angular Power Spectrum
• Deep Learning Complexity
• Activation Function Analysis