Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets

2026-07-18

AI Summary

This work investigates the mechanistic differences and asymptotic equivalence between Dropout and Random Gradient Masking (RaM) in deep ResNets. By leveraging asymptotic analysis in the joint limit of large depth and width, and integrating stochastic gradient dynamics with residual network modeling, the study rigorously establishes—for the first time—that Dropout and RaM induce identical training dynamics in the large-scale limit. This result encompasses various Dropout variants, including layer-wise Dropout and Stochastic Depth, thereby revealing a unified asymptotic behavior across diverse regularization strategies. The findings provide a principled theoretical foundation and a cohesive perspective for understanding regularization mechanisms in deep neural networks.

Abstract

Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.

Problem

• Dropout
• Random Gradient Masking
• ResNets
• asymptotic equivalence
• large-scale limit

Innovation

• asymptotic equivalence
• ResNets
• dropout
• Random Gradient Masking
• feature learning regime