GLOBE: Trajectory-Aligned Gradient Matching with Structured SparseOptimization for Coreset Selection

2026-08-03

AI Summary

This work addresses the computational and memory bottlenecks in on-device deep neural network training caused by large-scale data, where existing gradient-based coreset methods struggle to capture optimization dynamics and perform poorly on highly correlated samples. To overcome these limitations, the authors propose GLOBE, a novel framework that introduces gradient trajectory representations to characterize samples and aligns distributions via multi-order moment matching—specifically by aligning first-order means and second-order projected moments. GLOBE further induces structured sparsity at both group and sample levels through a combination of Group LASSO, Elastic Net, and non-negative budget constraints, while preserving class balance. Extensive experiments across six benchmark datasets and five architectures demonstrate that GLOBE significantly outperforms existing methods even at low retention rates, confirming its effectiveness for data-efficient learning.

Abstract

On-device training of deep neural networks is fundamentally constrained by the computational and memory costs of large-scale datasets. Coreset selection offers a practical solution by retaining only a compact subset of real training samples. However, existing gradient-based methods commonly rely on gradients computed at a single model snapshot and employ greedy or pursuit-based selection procedures, limiting their ability to capture evolving optimization dynamics and handle strongly correlated samples. We propose GLOBE (Gradient Local-Balanced Extraction), a trajectory-aligned coreset selection framework that formulates sample selection as a globally optimized sparse weighting problem. GLOBE represents each sample by a gradient trajectory constructed across multiple training checkpoints, thereby capturing its influence throughout different stages of optimization. To preserve the training behavior of the full dataset, we introduce a multi-order matching objective that jointly aligns the first-order mean and projected uncentered second-order moments of gradient trajectories. GLOBE further combines Group LASSO, Elastic Net regularization, and nonnegative budget constraints to induce group- and sample-level sparsity while stabilizing the weights of correlated trajectories. Finally, class-balanced Top-K selection maintains adequate category coverage under limited sampling budgets. Experiments across six benchmarks and five evaluation architectures demonstrate that GLOBE consistently outperforms existing coreset selection methods in downstream test accuracy, particularly at low retention ratios. These results highlight the effectiveness of combining dynamic gradient information, multi-order distribution matching, and structured sparsity for data-efficient learning.

Problem

• coreset selection
• on-device training
• gradient trajectory
• structured sparsity
• data-efficient learning

Innovation

• trajectory-aligned gradient matching
• structured sparse optimization
• coreset selection
• multi-order moment matching
• on-device training