reparameterization trick

This work addresses the high variance in stochastic gradient estimation, which often leads to unstable convergence, slow training, and limited generalization in deep learning. To mitigate this issue, the paper introduces survey sampling theory into optimization for the first time, proposing a model-assisted sampling framework. Treating the dataset as a finite population, the method leverages an auxiliary gradient prediction model to construct low-variance gradient estimators, seamlessly integrating with momentum-based optimizers like AdamW without altering their dynamics. Empirical results demonstrate that the approach significantly improves performance in 71–86% of experimental settings across synthetic and six benchmark datasets, achieving superior generalization in approximately half the usual training time. The framework unifies and generalizes both uniform sampling and efficient, auxiliary-information-driven sampling strategies.

Existing methods struggle to perform full-path statistical inference for gradient flow optimization trajectories, particularly lacking valid uncertainty quantification when stopping times are data-dependent or the path diverges. This work establishes a time-uniform statistical inference theory for gradient flows by modeling the deviation between empirical and population gradient flows as a continuous Gaussian process indexed over the non-negative real line. It presents the first uniform central limit theorem applicable across the entire optimization trajectory. Building on this foundation, the paper introduces an algorithm-aware covariance estimator that requires neither matrix inversion, resampling, nor data splitting, and which converges uniformly over time. The resulting confidence bands achieve asymptotically valid coverage, offering a theoretically rigorous and practically useful tool for path-level uncertainty quantification in gradient-based algorithms.

This work addresses the high variance and poor stability of feature attribution methods caused by gradient noise. To overcome these limitations, the authors propose a deterministic attribution framework based on linear interpolation path sampling. By establishing the equivalence between path sampling and weighted integrated gradients, the method reformulates stochastic estimation as a Riemann sum, enabling efficient and stable attribution computation. Theoretical analysis demonstrates that, under smooth models, the proposed approach improves the error convergence rate from $O(m^{-1/2})$ to $O(m^{-1})$. Moreover, under uniform sampling, it rigorously reduces attribution variance by one-third while preserving both linearity and implementation invariance.

To address the high computational cost and low efficiency of gradient updates in neural network training, this paper proposes StochGradAdam, a novel optimizer that integrates stochastic gradient sampling into the Adam framework. At each iteration, it computes gradients only over a randomly sampled subset of parameters, while incorporating dynamic weight updates and bias correction to prioritize critical gradient directions. Unlike standard Adam, StochGradAdam preserves adaptive learning rates and convergence stability while enhancing exploration of the loss landscape and mitigating gradient noise. Experimental results on image classification and semantic segmentation tasks demonstrate that StochGradAdam achieves comparable or superior accuracy to Adam with significantly fewer gradient evaluations. This leads to substantial acceleration in large-scale model training without compromising generalization performance, offering improved computational efficiency and robustness.

Gradient-based optimization of discrete categorical variables has long been hindered by either the high variance of score function estimators or the bias introduced by continuous relaxations. This work proposes a novel, training-free soft reparameterization approach by introducing denoising diffusion mechanisms to categorical variable modeling. Specifically, it constructs a differentiable sampler via a closed-form denoiser derived from a Gaussian noising process, enabling efficient gradient estimation with low bias. The method achieves optimization performance on par with or superior to existing techniques across multiple benchmark tasks, establishing a new paradigm for discrete optimization that is training-free, differentiable, and low-bias.

This work addresses the limited understanding of generalization in variance-reduced optimization methods, which have been predominantly analyzed through the lens of convergence. Focusing on algorithmic stability, the paper establishes sharp, data-dependent generalization error bounds for SVRG under both convex and strongly convex settings—the first such guarantees to date. The key innovation lies in decomposing the SVRG update into an SGD-like step and a zero-mean correction term, and in designing a novel Lyapunov function to handle the additional gradient terms introduced by the reference point. This framework yields optimal overall risk bounds in both settings and extends seamlessly to other variance-reduction algorithms like SAGA, thereby uncovering fundamental connections between optimization dynamics and generalization performance.

This work proposes a variational inference–based neural path estimation method for stochastic dynamical systems under partial observability, noisy measurements, and nonlinear dynamics. By constructing a controlled diffusion process, the approach maps a prior path measure to the posterior measure conditioned on observations, leveraging an observation-embedded representation to learn an optimal control policy. This enables joint learning of the underlying unknown stochastic differential equation (SDE) and its conditional path distribution. The method innovatively integrates the Zakai equation in path space with neural control, achieving, for the first time, end-to-end inference of posterior path measures. Experiments demonstrate that the framework accurately reconstructs system dynamics and efficiently infers posterior path distributions across multimodal, chaotic, and high-dimensional settings.

Bayesian parameter inference for complex stochastic simulators suffers from prohibitive computational cost due to intractable likelihoods—especially in high-dimensional parameter spaces or when simulator outputs are information-poor. To address this, we propose OptiBayes: a novel framework that reformulates stochastic simulation as a differentiable deterministic optimization problem. By leveraging gradient-based optimization, OptiBayes rapidly identifies high-posterior-density regions while substantially suppressing wasteful simulations in low-probability areas. Built upon an optimization–Monte Carlo hybrid paradigm and implemented end-to-end with JAX, it enables full vectorization and differentiability throughout the inference pipeline. Experiments demonstrate that OptiBayes matches or exceeds state-of-the-art methods in estimation accuracy while reducing runtime by one to two orders of magnitude. It further exhibits strong scalability, robustness to noise and model misspecification, and exceptional adaptability to weakly informative simulator outputs.

To address the high computational cost of gradient computation in neural network training, this paper proposes an unbiased and efficient approximate gradient estimation method. Our approach introduces three key innovations: (1) a lightweight gradient predictor inspired by the Neural Tangent Kernel theory, enabling rapid gradient approximation in forward mode; (2) control variates to rigorously guarantee estimator unbiasedness and controllable variance; and (3) integration with a linear approximation model, validated end-to-end on Vision Transformers. Experiments on benchmarks including ImageNet demonstrate that our method reduces training FLOPs by up to 38% while maintaining convergence speed and final accuracy comparable to full-gradient training. The proposed framework offers a novel paradigm for efficient large-model training without sacrificing optimization fidelity or generalization performance.

This work addresses the challenge of optimizing objectives with intractable gradients—such as maximum marginal likelihood estimation and generative model fine-tuning—by introducing, for the first time, a sequential Monte Carlo (SMC) sampler into a gradient-free optimization framework. The proposed approach replaces the conventional inner-loop sampling procedure to efficiently approximate stochastic gradients, substantially reducing computational overhead while preserving optimization performance. A theoretical convergence analysis is provided to support the method’s validity. Empirical evaluations across multiple energy-based models on reward tuning tasks demonstrate that the proposed algorithm achieves significant acceleration without compromising, and in some cases even improving, optimization quality.