Innovation
Methods, ideas, or system contributions that make the work stand out.
Benign Overfitting
General Norms
Sub-Gaussian Distributions
Dual Optimization Geometry
Minimum-p-norm Interpolation
Benign Overfitting for General Norms and Distributions
🤖 AI Summary
This study addresses the lack of theory for benign overfitting in linear regression under non-Euclidean norms and the limitations of Gaussian assumptions by establishing a unified analytical framework for general norms and sub-Gaussian distributions. By integrating dual optimization geometry, concentration inequalities, and the central limit theorem, we rigorously prove that norms with p>1 exhibit benign overfitting under non-Gaussian data, while revealing that analogous results for the 1-norm inherently rely on Gaussianity. This work transcends existing theoretical boundaries by deriving general overfitting conditions for arbitrary norms and correcting prevailing misconceptions regarding the 1-norm. Ultimately, it deepens the understanding of how the geometric properties of modern optimizers influence generalization mechanisms.
📝 Abstract
Understanding why predictors can generalize despite interpolating noisy training data is a central puzzle in machine learning. Most work on such"benign overfitting"studies minimum-2-norm linear regression, reflecting the inductive bias of gradient descent. However, modern optimizers such as Adam and Muon use non-Euclidean update geometries, favoring solutions associated with other norms. Analyzing regression for non-Euclidean norms is substantially more difficult, with known results essentially limited to Gaussians. In this paper, we develop a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions. As a special case, we prove that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions. Perhaps surprisingly, for the 1-norm, benign overfitting does not hold in general for well-behaved (but non-Gaussian) distributions, showing that existing positive 1-norm results rely crucially on Gaussianity. Our proof analyzes the geometry of the dual optimization problem, using concentration and central limit tools to show it is approximately Euclidean in many high-dimensional cases.
🔎 Similar Papers
No similar papers found.
