🤖 AI Summary
This study investigates the origins of gradient explosion and vanishing in deep neural networks, with a focus on how residual connections influence gradient propagation dynamics. Drawing upon multiplicative ergodic theory, the work introduces—for the first time—the characterization of Lyapunov exponents due to Furstenberg–Kifer, integrating tools from random matrix theory and modeling the residual architecture to systematically analyze the evolution of gradients. The analysis reveals that residual connections effectively compress the spread of the Lyapunov spectrum, thereby substantially mitigating gradient instability. These findings offer crucial theoretical grounding for the principled design of deep networks, elucidating why residual structures enhance trainability in very deep models.
📝 Abstract
The well known phenomenon of exploding and vanishing gradients in deep neural networks is analyzed using multiplicative ergodic theory. The effect of adding a residual connection is explained in this context. Specifically, a characterization of Liapunov exponents due to Furstenberg and Kifer is exploited in order to make a precise statement about the Liapunov spectrum and the effect of residual connections on it.