🤖 AI Summary
This work addresses the challenges of non-convex optimization and theoretical gaps in shallow neural network training by reframing the discrete parameter learning problem as a continuous variational problem over parameter densities in a weighted Sobolev space. The authors propose a globally well-posed and stable optimization framework grounded in λ-convex functionals, innovatively integrating elliptic regularity theory with variational analysis to bypass conventional iterative optimization. Instead, the optimal parameter density is obtained directly by solving a single linear system, thereby unifying the neural tangent kernel (NTK) and feature learning perspectives. Theoretically, the solution converges to the continuous optimum at a rate of O(1/N), and the generalization error is bounded by O(1/α), significantly enhancing both training efficiency and theoretical interpretability.
📝 Abstract
Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of $λ$-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost $C^3$ regularity.
Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of $1/α$ relative to the regularization parameter, and prove that finite-width networks of size $N$ achieve the continuum optimum at an $O(1/N)$ rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.