🤖 AI Summary
This paper investigates the optimization and statistical behavior of Randomly Weighted Gradient Descent (RWGD) in linear regression. We analyze RWGD under general continuous weight distributions—including SGD and importance sampling as special cases—and establish the first non-asymptotic bounds on the convergence of both first- and second-order moments. We uncover an implicit regularization mechanism: the stationary distribution induced by gradient noise is characterized via geometric moment contraction, and we prove that weight design fundamentally governs the bias–variance trade-off. Crucially, we identify that rapidly convergent weight schemes inherently sacrifice statistical accuracy, and we provide the first quantitative characterization of the fundamental trade-off between convergence rate and estimation precision. Our results yield a unified analytical framework and concrete design principles for developing robust and efficient learning algorithms.
📝 Abstract
We analyze gradient descent with randomly weighted data points in a linear regression model, under a generic weighting distribution. This includes various forms of stochastic gradient descent, importance sampling, but also extends to weighting distributions with arbitrary continuous values, thereby providing a unified framework to analyze the impact of various kinds of noise on the training trajectory. We characterize the implicit regularization induced through the random weighting, connect it with weighted linear regression, and derive non-asymptotic bounds for convergence in first and second moments. Leveraging geometric moment contraction, we also investigate the stationary distribution induced by the added noise. Based on these results, we discuss how specific choices of weighting distribution influence both the underlying optimization problem and statistical properties of the resulting estimator, as well as some examples for which weightings that lead to fast convergence cause bad statistical performance.