Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization

2026-07-17

AI Summary

This work addresses a key limitation of standard Sharpness-Aware Minimization (SAM), which allocates a global perturbation budget based on instantaneous mini-batch gradient norms, rendering it sensitive to noise and unable to accurately capture the true sensitivity of individual parameter blocks. To overcome this, the authors propose GEAR-SAM, which introduces gradient energy—a lightweight proxy for curvature sensitivity—by maintaining an exponential moving average of squared gradients per parameter block. Within a fixed global perturbation radius, GEAR-SAM dynamically reallocates block-wise budgets via closed-form constrained optimization. Notably, it avoids computationally expensive Hessian-vector products or explicit Fisher matrix estimation, incurring only scalar state overhead per block. Experiments demonstrate that GEAR-SAM consistently enhances generalization and robustness across image classification, transfer learning, and learning with noisy labels, while remaining compatible with diverse network architectures.

Abstract

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss in a local parameter neighborhood. Standard SAM implicitly allocates its global perturbation budget across parameter blocks according to instantaneous minibatch gradient norms. Such an allocation can be noisy and may not reflect the sensitivity that blocks accumulate throughout training. We propose Gradient-Energy Adaptive Radius SAM (GEAR-SAM), which maintains an exponential moving average (EMA) of squared block gradients as a lightweight, curvature-related sensitivity signal and allocates the fixed SAM budget through a closed-form constrained optimization. GEAR-SAM preserves the global SAM radius, requires no Hessian-vector products or explicit Fisher estimation, and adds only scalar state beyond SAM. Experiments on image classification, transfer learning, noisy-label learning, and partition studies demonstrate improved generalization and robustness across architectures and tasks. More broadly, GEAR-SAM provides a dynamic view of sharpness-aware optimization: a fixed perturbation budget should be redistributed as the sensitivity of functional network blocks evolves during training.

Problem

• Sharpness-Aware Minimization
• perturbation budget allocation
• gradient noise
• parameter sensitivity
• generalization

Innovation

• Sharpness-Aware Minimization
• Gradient Energy
• Block-wise Perturbation
• Adaptive Budget Allocation
• Exponential Moving Average