Robustness Emerges Early in Training Dynamics, but Is Not Preserved

2026-08-05

AI Summary

This work addresses the degradation of robustness in deep neural networks during standard training, where although robust representations and flat loss landscapes emerge spontaneously in early epochs, these favorable properties deteriorate as training progresses, leading to insufficient robustness against natural perturbations. To counteract this issue, the paper proposes a training-dynamic intervention framework that requires no architectural modifications or additional parameters. The approach introduces two novel strategies—Early Phase Stabilization (EPS) and Asymmetric Weight Rollback (AWR)—to systematically preserve or restore the robust priors established in early training stages. Extensive experiments demonstrate that this method consistently enhances model robustness, transferability, and dynamic adaptability across diverse benchmarks and vision tasks.

Abstract

Robustness to natural corruptions remains a fundamental challenge for deep neural networks. In this paper, we identify a robustness fading phenomenon where shallow layers spontaneously develop robust representations and flat loss landscapes in early training, yet these properties are not preserved during standard convergence. To address this, we propose a framework that performs strategic interventions on training dynamics to stabilize the empirically identified early-emergent robust priors. Our approach includes two parameter-free strategies: Early-Phase Stabilization~(EPS) and Asymmetric Weight Reversion~(AWR), which stabilize or recover robust shallow configurations without modifying the model architecture or introducing learnable parameters. Extensive experiments demonstrate the efficacy of our framework across various benchmarks and architectures, yielding significant gains in downstream transfer, dynamic adaptation, and diverse computer vision applications.

Problem

• robustness
• natural corruptions
• training dynamics
• loss landscape
• deep neural networks

Innovation

• robustness fading
• early-phase stabilization
• asymmetric weight reversion
• training dynamics
• parameter-free intervention