🤖 AI Summary
Traditional model regularization relies on manual hyperparameter tuning to balance model complexity and overfitting. This paper proposes Cross-Reg, a novel cross-validation-inspired regularization framework that for the first time directly leverages validation-set gradients to dynamically optimize regularization strength: training data drive feature learning, while validation data adaptively govern model complexity—eliminating manual intervention entirely. Built upon gradient-based updates with controlled noise injection, Cross-Reg converges efficiently to a near-optimal solution approximating full cross-validation, all within a single training run. It automatically discovers architecture- and task-adapted regularization patterns and natively supports extensions such as data augmentation and uncertainty calibration. Experiments demonstrate substantial improvements in generalization performance and strong robustness to label noise, establishing a new data-driven paradigm for adaptive regularization.
📝 Abstract
Model regularization requires extensive manual tuning to balance complexity against overfitting. Cross-regularization resolves this tradeoff by directly adapting regularization parameters through validation gradients during training. The method splits parameter optimization - training data guides feature learning while validation data shapes complexity controls - converging provably to cross-validation optima. When implemented through noise injection in neural networks, this approach reveals striking patterns: unexpectedly high noise tolerance and architecture-specific regularization that emerges organically during training. Beyond complexity control, the framework integrates seamlessly with data augmentation, uncertainty calibration and growing datasets while maintaining single-run efficiency through a simple gradient-based approach.