🤖 AI Summary
This work addresses the limited generalization and robustness of deep models on clean, corrupted, and out-of-distribution (OOD) data by proposing an interleaved noise injection training strategy. During training, noisy and clean samples are alternately presented, complemented by gradient norm stabilization to enhance both feature exploration and preservation. Theoretical analysis reveals that impulsive noise is equivalent to Jacobian regularization, while Gaussian noise corresponds to curvature penalization; together, they mitigate failure modes induced by the model’s inductive bias. The approach incurs negligible computational overhead and significantly improves the robustness of both ResNet and Vision Transformer (ViT) architectures on CIFAR-100-C, ImageNet-C, and ImageNet-R benchmarks. Moreover, it seamlessly integrates with existing data augmentation techniques, yielding complementary gains.
📝 Abstract
Noise injection is a well-known technique in stochastic optimization. We report its surprising effectiveness with an interleaved (on-off-on-off...) rather than the usual monotonic decay schedule. We present a theoretical analysis of noise injection, which confirms that corruption by impulse noise approximates a Jacobian regularization, whereas Gaussian noise acts as a curvature penalty. This regularization behavior has been invoked to explain why noise injection increases model robustness. But the interleaved nature of our proposed schedule produces superior results even for the optimization objective: mixing phases of noisy data permits the optimizer to escape local minima and increase exploration without the risk of catastrophically forgetting the important features from the clean data. To stabilize this training scheme against the rapid changes of the loss when switching between clean and noisy data, we introduce a gradient-norm stabilization technique that scales noisy updates based on clean gradient magnitudes. We compare this method with other common augmentation methods and find substantial improvements in corruption tolerance and robustness to real-world distribution shifts on CIFAR-100-C, ImageNet-C, and ImageNet-R for ResNet and ViT architectures, with the best results being achieved by stacking our method on top of other augmentations. Through saliency and attention maps we show that the effect of interleaved noise injection stems from penalizing the failure modes encouraged by the inductive bias of the models: impulse noise works against the locality bias of convolutional (ResNet) architectures, and Gaussian noise reduces the tendency of attention-based models to pick up large-scale spurious features. Interleaved noise injection is therefore an effective tool to improve the test performance on clean, noisy, and out-of-distribution data at essentially zero computational cost.