Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

2026-08-04

AI Summary

This work investigates the implicit bias of gradient descent in the early phase of training logistic regression models, revealing that the parameter vector achieves weak alignment with the max-margin direction in far fewer iterations than required by asymptotic analyses. By directly analyzing the radial and tangential dynamics of the parameter vector in conjunction with the geometric structure of the data, the study bypasses conventional asymptotic approaches and establishes the first non-asymptotic theoretical bound for early alignment: alignment occurs within $O(\exp(\exp(-\delta)))$ iterations, and this bound is tight. The result rigorously characterizes the positive correlation between the parameter vector and the max-margin direction during early training, offering a dynamical explanation for the empirical observation that longer training often improves generalization.

Abstract

Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns. A typical instance is the max-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions. Even after having a given dataset separated, the parameter vector continues to evolve towards the max-margin direction asymptotically along the gradient descent dynamics. This phenomenon corroborates a frequent empirical observation of "train longer, generalize better." However, the max-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max-margin direction positively (though not exactly) within considerably fewer iterations than the asymptotic rate. By shedding another light on this classical problem, this work aims to understand the mechanism of this early-stage alignment phenomenon. Our theoretical results demonstrate that the parameter vector weakly aligns with the max-margin direction within $O(\exp(\exp(-δ)))$ iterations, where $δ>0$ is the permissible alignment error, which is shown to be tight. By tracking the radial and tangential flows, our proof operates on the alignment dynamics directly with dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak alignment.

Problem

• implicit bias
• gradient descent
• early-stage dynamics
• max-margin
• non-asymptotic

Innovation

• implicit bias
• gradient descent
• max-margin
• non-asymptotic analysis
• alignment dynamics