🤖 AI Summary
This work investigates the failure of generalization in empirical risk minimization (ERM) for stochastic convex optimization when the sample size scales linearly with the problem dimension. By constructing a specific convex learning instance, the authors provide the first proof that ERM can be uniquely and necessarily overfitted under this regime, thereby resolving an open problem posed by Feldman. The analysis is further extended to approximate ERM and gradient descent algorithms. Combining tools from convex optimization theory, probabilistic constructions, and dynamical systems analysis, the study establishes a lower bound of Ω(√(ηT/m^{1.5})) on the generalization error of gradient descent, significantly narrowing the exponential gap between the previously known upper bound of O(ηT/m) and the true behavior of the algorithm.
📝 Abstract
We study the sample complexity of the best-case Empirical Risk Minimizer in the setting of stochastic convex optimization. We show that there exists an instance in which the sample size is linear in the dimension, learning is possible, but the Empirical Risk Minimizer is likely to be unique and to overfit. This resolves an open question by Feldman. We also extend this to approximate ERMs. Building on our construction we also show that (constrained) Gradient Descent potentially overfits when horizon and learning rate grow w.r.t sample size. Specifically we provide a novel generalization lower bound of $\Omega\left(\sqrt{\eta T/m^{1.5}}\right)$ for Gradient Descent, where $\eta$ is the learning rate, $T$ is the horizon and $m$ is the sample size. This narrows down, exponentially, the gap between the best known upper bound of $O(\eta T/m)$ and existing lower bounds from previous constructions.