From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning

2026-07-24

AI Summary

This work addresses the challenge in long-tailed learning where head classes dominate training, causing tail classes to converge to sharp minima in the loss landscape and suffer degraded generalization. The authors propose Sharpness-Guided Sampling (SGS), a novel approach that actively shapes the geometry of the loss landscape from the sampling perspective. By dynamically adjusting minibatch sampling probabilities based on cumulative class frequencies and exponential moving average (EMA) estimates of sharpness, SGS establishes a frequency-sharpness feedback mechanism that steers optimization toward balanced, flat minima—without requiring class perturbations or additional backward passes. Theoretically, the method is supported by a sampling-aware PAC-Bayes analysis and continuous-time stochastic differential equation modeling. Experiments demonstrate that SGS improves tail accuracy by 10.85 points (overall +3.56) on CIFAR-100-LT and by 6.59 points (overall +1.20) on ImageNet-LT, with only 1.02× the training cost of SAM.

Abstract

Long-tailed learning couples two sources of poor generalization: head classes dominate training exposure, while under-represented classes often converge to sharper regions of the loss landscape. Conventional re-sampling addresses the former without considering geometry, whereas existing long-tailed sharpness-aware minimization (SAM) methods modify losses or perturbations only after biased mini-batches have been drawn. We introduce Sharpness-Guided Equilibrium Sampling (SGS), which treats the sampling distribution as an active control variable for optimization geometry. SGS dynamically adjusts subsequent mini-batches by increasing the sampling probability of less frequently sampled classes while suppressing classes with large SAM-induced loss changes, using only cumulative class counts and EMA sharpness estimates obtained from the standard SAM update, without class-wise perturbations or additional backward passes. We characterize this sampling process through a continuous-time stochastic differential equation and a sampling-dependent PAC-Bayes analysis, explaining how frequency-sharpness feedback can move training toward a more balanced flatness profile. On CIFAR-100 LT with an imbalance ratio of 100, SGS-SAM improves Focal-SAM by 10.85 points in tail accuracy and 3.56 points overall. On ImageNet-LT, it improves ImbSAM by 6.59 points on tail classes and 1.20 points overall. Its training time is only $1.02\times$ that of vanilla SAM. Beyond these gains, SGS establishes a sampling-side route to loss-landscape control, suggesting that future long-tailed methods can jointly regulate data exposure and optimization geometry rather than treating either as fixed.

Problem

• long-tailed learning
• sharpness-aware minimization
• loss landscape
• class imbalance
• flat minima

Innovation

• Sharpness-Guided Sampling
• Long-Tailed Learning
• Flat Minima
• Equilibrium Sampling
• Loss Landscape Geometry