🤖 AI Summary
To address the slow convergence of stochastic second-order methods for large-scale finite-sum optimization, this paper proposes the Stochastic Variance-Reduced Newton (SVRN) method—the first to integrate variance reduction into a stochastic second-order framework. SVRN combines subsampled Newton updates, Hessian sketching, and large-batch parallel computation, employing a unit step size to ensure stable convergence. Theoretically, SVRN achieves a data-pass complexity of $O(log(1/varepsilon)/log n)$, improving upon conventional stochastic second-order methods by a factor of $O(alpha log n)$; notably, this acceleration ratio increases with sample size $n$. Empirically, under large-batch settings, SVRN significantly outperforms first-order variance-reduced methods such as SAGA and SVRG.
📝 Abstract
Stochastic variance reduction has proven effective at accelerating first-order algorithms for solving convex finite-sum optimization tasks such as empirical risk minimization. Incorporating second-order information has proven helpful in further improving the performance of these first-order methods. Yet, comparatively little is known about the benefits of using variance reduction to accelerate popular stochastic second-order methods such as Subsampled Newton. To address this, we propose Stochastic Variance-Reduced Newton (SVRN), a finite-sum minimization algorithm that provably accelerates existing stochastic Newton methods from $O(alphalog(1/epsilon))$ to $Oig(frac{log(1/epsilon)}{log(n)}ig)$ passes over the data, i.e., by a factor of $O(alphalog(n))$, where $n$ is the number of sum components and $alpha$ is the approximation factor in the Hessian estimate. Surprisingly, this acceleration gets more significant the larger the data size $n$, which is a unique property of SVRN. Our algorithm retains the key advantages of Newton-type methods, such as easily parallelizable large-batch operations and a simple unit step size. We use SVRN to accelerate Subsampled Newton and Iterative Hessian Sketch algorithms, and show that it compares favorably to popular first-order methods with variance~reduction.