🤖 AI Summary
This paper addresses uncertainty propagation in feedforward neural networks under stochastic input perturbations, with a focus on architectures employing the Leaky ReLU activation function. We propose a high-accuracy analytical method: first, piecewise linearization of Leaky ReLU preserves its essential nonlinearity; second, a Gaussian Copula-based joint distribution surrogate model is constructed to yield closed-form expressions for the output probability density function and arbitrary-order statistical moments. The approach circumvents the prohibitive computational cost of Monte Carlo simulation while achieving excellent agreement between theoretical predictions and numerical experiments. Its robustness to large input perturbations is further validated in modeling nonlinear integral-differential operators within polynomial function spaces. The core contribution is the first rigorous, analytically tractable, high-fidelity, and robust theoretical framework for uncertainty propagation in Leaky ReLU networks.
📝 Abstract
We develop new uncertainty propagation methods for feed-forward neural network architectures with leaky ReLu activation functions subject to random perturbations in the input vectors. In particular, we derive analytical expressions for the probability density function (PDF) of the neural network output and its statistical moments as a function of the input uncertainty and the parameters of the network, i.e., weights and biases. A key finding is that an appropriate linearization of the leaky ReLu activation function yields accurate statistical results even for large perturbations in the input vectors. This can be attributed to the way information propagates through the network. We also propose new analytically tractable Gaussian copula surrogate models to approximate the full joint PDF of the neural network output. To validate our theorical results, we conduct Monte Carlo simulations and a thorough error analysis on a multi-layer neural network representing a nonlinear integro-differential operator between two polynomial function spaces. Our findings demonstrate excellent agreement between the theoretical predictions and Monte Carlo simulations.