variance reduction

This work addresses the limitations of existing stochastic variance reduction methods, whose analyses are often confined to expectation-level guarantees and tied to specific algorithms, thus failing to capture the structural trade-offs underlying reliability. The paper proposes a unified variance-reduced estimation framework that integrates memory retention, reset probability, and iterative moving correction, applicable in both Euclidean and non-Euclidean spaces. Its key contributions include the first dimension-free vector-valued Freedman inequality in smooth normed spaces, which—combined with martingale theory and mirror descent—enables a unified high-probability error analysis. Furthermore, the framework extends variance reduction to stochastic optimization with expectation constraints, achieving logarithmic confidence-dependent complexity for unconstrained mirror descent and improving the oracle complexity for constrained problems from ~O(ε⁻⁴) to ~O(ε⁻³).

This paper addresses the computational bottleneck of variance-reduced stochastic optimization algorithms (e.g., SVRG, SARAH) in large-scale machine learning—their reliance on expensive full-gradient evaluations. We propose a novel, full-gradient-free variance reduction method that integrates random reshuffling with the gradient caching mechanism of SAG/SAGA, augmented by recursive gradient updates and a new analytical framework for variance control without full gradients. Theoretically, our method achieves the same convergence rate as classical reshuffling in non-convex settings and, for the first time among full-gradient-free methods, attains a superior rate in strongly convex settings. Empirically, it accelerates training by 30–50% on large-scale datasets while reducing memory overhead by 90%. Our key contribution is the first full-gradient-free algorithm achieving SVRG-/SARAH-level variance reduction, thereby eliminating the need for periodic full-gradient computations entirely.

To address the slow convergence of stochastic second-order methods for large-scale finite-sum optimization, this paper proposes the Stochastic Variance-Reduced Newton (SVRN) method—the first to integrate variance reduction into a stochastic second-order framework. SVRN combines subsampled Newton updates, Hessian sketching, and large-batch parallel computation, employing a unit step size to ensure stable convergence. Theoretically, SVRN achieves a data-pass complexity of $O(log(1/varepsilon)/log n)$, improving upon conventional stochastic second-order methods by a factor of $O(alpha log n)$; notably, this acceleration ratio increases with sample size $n$. Empirically, under large-batch settings, SVRN significantly outperforms first-order variance-reduced methods such as SAGA and SVRG.

Variance reduction (VR) optimization algorithms face challenges in large language model (LLM) training—including limited applicability, slow convergence, and instability. To address these issues, we propose MARS, a unified optimization framework that tightly integrates VR techniques with preconditioned gradient methods. MARS introduces a scaled stochastic recursive momentum mechanism and an adaptive learning rate strategy. It is instantiated into three variants compatible with AdamW, Lion, and Shampoo, thereby establishing the first theoretical connection between VR methods and mainstream adaptive optimizers. Empirical evaluation on GPT-2 training demonstrates that MARS achieves up to 25% faster convergence than AdamW while exhibiting enhanced training stability. The implementation is publicly available.

Standard SVRG suffers from degraded performance in deep neural network training due to its fixed variance-reduction strength. To address this, we propose α-SVRG, which introduces a linearly decaying multiplicative coefficient α into the SVRG framework to dynamically modulate the weight of the variance-reduction term—without altering the gradient update structure. This design is the first to empirically identify and formally model the critical principle that optimal variance-reduction strength should adaptively decrease with both network depth and training progression. Extensive experiments across multiple CNN architectures (e.g., ResNet, VGG) and standard image classification benchmarks (CIFAR-10/100, ImageNet) demonstrate that α-SVRG consistently achieves lower training loss and outperforms both vanilla SVRG and SGD baselines. Our method effectively resolves the well-documented failure of SVRG in deep learning settings, offering improved convergence stability and generalization.

This work addresses the limited understanding of generalization in variance-reduced optimization methods, which have been predominantly analyzed through the lens of convergence. Focusing on algorithmic stability, the paper establishes sharp, data-dependent generalization error bounds for SVRG under both convex and strongly convex settings—the first such guarantees to date. The key innovation lies in decomposing the SVRG update into an SGD-like step and a zero-mean correction term, and in designing a novel Lyapunov function to handle the additional gradient terms introduced by the reference point. This framework yields optimal overall risk bounds in both settings and extends seamlessly to other variance-reduction algorithms like SAGA, thereby uncovering fundamental connections between optimization dynamics and generalization performance.

This work addresses the high variance in stochastic gradient estimation, which often leads to unstable convergence, slow training, and limited generalization in deep learning. To mitigate this issue, the paper introduces survey sampling theory into optimization for the first time, proposing a model-assisted sampling framework. Treating the dataset as a finite population, the method leverages an auxiliary gradient prediction model to construct low-variance gradient estimators, seamlessly integrating with momentum-based optimizers like AdamW without altering their dynamics. Empirical results demonstrate that the approach significantly improves performance in 71–86% of experimental settings across synthetic and six benchmark datasets, achieving superior generalization in approximately half the usual training time. The framework unifies and generalizes both uniform sampling and efficient, auxiliary-information-driven sampling strategies.

This work addresses the issue in non-convex stochastic optimization under Blum–Gladyshev (BG) noise, where gradient variance grows quadratically with the distance from the initialization point. The authors propose a normalized momentum stochastic gradient descent method combined with a variance-reduced STORM algorithm. Under standard and generalized smoothness assumptions, they establish the first convergence guarantees for normalized methods in unbounded domains with fixed batch sizes and without explicit anchoring under BG-0 noise. Their analysis reveals that generalized smoothness has a rate-neutral effect on normalized momentum methods. Theoretically, the normalized momentum method achieves an $O(\varepsilon^{-6})$ complexity, while the variance-reduced STORM attains the optimal $O(\varepsilon^{-4})$ rate under sharp initialization and $O(\varepsilon^{-(4+\alpha)})$ for $\alpha \in (0,1)$ or $O(\varepsilon^{-5})$ when $\alpha = 1$ under generalized smoothness.

This work addresses the high computational cost incurred when using pretrained diffusion models as fixed teachers in downstream tasks, where gradient estimation relies on high-variance Monte Carlo expectations. To mitigate this, the authors propose the CARV framework, which introduces a computation-aware variance accounting mechanism. By reusing expensive upstream computations and integrating timestep importance sampling with a stratified inverse CDF construction, CARV yields a low-variance stratified Monte Carlo estimator. Empirical results demonstrate 2–3× effective computational gains in text-to-3D distillation and data attribution tasks, along with an order-of-magnitude variance reduction in single-step distillation. However, no significant improvement in FID is observed, suggesting that variance is no longer the primary performance bottleneck.

Existing single-step lookahead Bayesian optimization methods rely on complex approximations and lack theoretical regret guarantees. This work proposes the Optimum Variance Reduction (OVR) approach, which requires only posterior sampling and Monte Carlo approximation, augmented with a regularization mechanism to encourage exploration. For the first time, we establish a global uniform error bound for single-step lookahead Bayesian optimization based on Monte Carlo approximation and prove that the regularized OVR algorithm achieves an asymptotically vanishing upper bound on Bayesian expected simple regret. Both theoretical analysis and numerical experiments corroborate the convergence and effectiveness of the proposed method.