Stress-Aware Resilient Neural Training

2025-07-31

AI Summary

Deep neural networks often converge to sharp minima during training stagnation, degrading generalization performance. Method: This paper proposes a stress-aware learning paradigm inspired by material fatigue mechanisms—introducing elastic/plastic deformation concepts into neural training for the first time. It constructs internal stress signals to dynamically detect optimization plateaus and designs an adaptive noise injection strategy to facilitate escape from sharp minima. Based on this, we propose the Plastic Deformation Optimizer (PDO), which integrates loss and accuracy stagnation signals to enable elastic response. Contribution/Results: Extensive experiments across six network architectures, four baseline optimizers, and seven vision benchmarks demonstrate that PDO significantly improves model robustness and generalization while incurring negligible computational overhead.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

This paper introduces Stress-Aware Learning, a resilient neural training paradigm in which deep neural networks dynamically adjust their optimization behavior - whether under stable training regimes or in settings with uncertain dynamics - based on the concept of Temporary (Elastic) and Permanent (Plastic) Deformation, inspired by structural fatigue in materials science. To instantiate this concept, we propose Plastic Deformation Optimizer, a stress-aware mechanism that injects adaptive noise into model parameters whenever an internal stress signal - reflecting stagnation in training loss and accuracy - indicates persistent optimization difficulty. This enables the model to escape sharp minima and converge toward flatter, more generalizable regions of the loss landscape. Experiments across six architectures, four optimizers, and seven vision benchmarks demonstrate improved robustness and generalization with minimal computational overhead. The code and 3D visuals will be available on GitHub: https://github.com/Stress-Aware-Learning/SAL.

Problem

• Dynamic optimization adjustment in neural networks under stress
• Escaping sharp minima for flatter, generalizable loss landscapes
• Improving robustness and generalization with minimal computational cost

Innovation

• Dynamic optimization adjustment via stress signals
• Plastic Deformation Optimizer with adaptive noise
• Escaping sharp minima for flatter loss landscapes