Genus expansion for non-linear random matrix ensembles with applications to neural networks

2024-07-11

AI Summary

This work investigates the asymptotic behavior of randomly initialized neural networks in the wide-limit regime, addressing three core problems: (i) the convergence rate to a Gaussian process, (ii) the Frobenius-norm convergence rate of the neural tangent kernel (NTK) to its deterministic limit, and (iii) higher-order moments of the limiting spectral distribution of the Jacobian matrix. Methodologically, we introduce a unified analytical framework: (i) extending the Faà di Bruno formula to multivariate composition for linearizing activation function effects; (ii) pioneering the application of genus expansion to neural network initialization analysis; and (iii) developing a graph-indexed stochastic multilinear mapping expansion. We rigorously establish Gaussian process convergence as width tends to infinity; quantify the Frobenius convergence rate of the NTK; and derive, for the first time, closed-form expressions for arbitrary-order moments of the limiting Jacobian spectrum—generalized to sparse and non-Gaussian weight settings.

Technology Category

Machine Learning: Kernel MethodsNatural Language Processing: Learning & Optimization for NLPCognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsWeb Mining and Content Analysis: Models for Web evolutionSocial Networks and Social Media: Social media analysis through the lenses of networks

Abstract

We present a unified approach to studying certain non-linear random matrix ensembles and associated random neural networks at initialization. This begins with a novel series expansion for neural networks which generalizes Fa'a di Bruno's formula to an arbitrary number of compositions. The role of monomials is played by random multilinear maps indexed by directed graphs, whose edges correspond to random matrices. Crucially, this expansion linearizes the effect of the activation functions, allowing for the direct application of Wick's principle and the genus expansion technique. As an application, we prove several results about neural networks with random weights. We first give a new proof of the fact that they converge to Gaussian processes as their width tends to infinity. Secondly, we quantify the rate of convergence of the Neural Tangent Kernel to its deterministic limit in Frobenius norm. Finally, we compute the moments of the limiting spectral distribution of the Jacobian (only the first two of which were previously known), expressing them as sums over non-crossing partitions. All of these results are then generalised to the case of neural networks with sparse and non-Gaussian weights, under moment assumptions.

Problem

• Develops a unified approach for non-linear random matrix ensembles and neural networks
• Quantifies convergence rate of Neural Tangent Kernel in Frobenius norm
• Computes moments of Jacobian's spectral distribution for sparse non-Gaussian weights

Innovation

• Generalized series expansion for neural networks
• Linearized activation via random multilinear maps
• Genus expansion technique with Wick's principle