🤖 AI Summary
Standard SVRG suffers from degraded performance in deep neural network training due to its fixed variance-reduction strength. To address this, we propose α-SVRG, which introduces a linearly decaying multiplicative coefficient α into the SVRG framework to dynamically modulate the weight of the variance-reduction term—without altering the gradient update structure. This design is the first to empirically identify and formally model the critical principle that optimal variance-reduction strength should adaptively decrease with both network depth and training progression. Extensive experiments across multiple CNN architectures (e.g., ResNet, VGG) and standard image classification benchmarks (CIFAR-10/100, ImageNet) demonstrate that α-SVRG consistently achieves lower training loss and outperforms both vanilla SVRG and SGD baselines. Our method effectively resolves the well-documented failure of SVRG in deep learning settings, offering improved convergence stability and generalization.
📝 Abstract
Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson&Zhang (2013), is a theoretically compelling optimization method. However, as Defazio&Bottou (2019) highlights, its effectiveness in deep learning is yet to be proven. In this work, we demonstrate the potential of SVRG in optimizing real-world neural networks. Our analysis finds that, for deeper networks, the strength of the variance reduction term in SVRG should be smaller and decrease as training progresses. Inspired by this, we introduce a multiplicative coefficient $alpha$ to control the strength and adjust it through a linear decay schedule. We name our method $alpha$-SVRG. Our results show $alpha$-SVRG better optimizes neural networks, consistently reducing training loss compared to both baseline and the standard SVRG across various architectures and image classification datasets. We hope our findings encourage further exploration into variance reduction techniques in deep learning. Code is available at https://github.com/davidyyd/alpha-SVRG.
