🤖 AI Summary
This work investigates tight upper and lower bounds on the Lipschitz constant of random ReLU neural networks to characterize their adversarial robustness. We propose a spectral norm estimation framework grounded in probabilistic analysis, random matrix theory, and layerwise propagation modeling of the ReLU nonlinearity, specifically under generalized He initialization—including both weights and symmetric biases. Our analysis yields the first matching upper and lower bounds under this initialization: for shallow networks, we achieve constant-factor precision; for deep, wide networks with fixed depth, we obtain bounds tight up to logarithmic factors in width—substantially improving upon prior results suffering from polynomial-width dependencies. These theoretical guarantees establish a rigorous benchmark for worst-case robustness of randomly initialized ReLU networks.
📝 Abstract
Empirical studies have widely demonstrated that neural networks are highly sensitive to small, adversarial perturbations of the input. The worst-case robustness against these so-called adversarial examples can be quantified by the Lipschitz constant of the neural network. In this paper, we study upper and lower bounds for the Lipschitz constant of random ReLU neural networks. Specifically, we assume that the weights and biases follow a generalization of the He initialization, where general symmetric distributions for the biases are permitted. For shallow neural networks, we characterize the Lipschitz constant up to an absolute numerical constant. For deep networks with fixed depth and sufficiently large width, our established upper bound is larger than the lower bound by a factor that is logarithmic in the width.
