real-analytic function theory

This paper investigates the optimization landscape of shallow single-hidden-layer analytic neural networks under mean-squared-error loss for regression. Specifically, it characterizes the strong convexity of neighborhoods around local minima and its implications for the convergence rate of first-order optimizers. Methodologically, the analysis integrates differential topology and Morse theory with stochastic modeling of regression problems and geometric decomposition of the parameter space. The key contribution is the first rigorous proof that, over the efficient parameter regime—i.e., the set of functions realizable only with the given number of neurons—the loss function is almost surely a Morse function for almost all regression tasks; consequently, all local minima possess strongly convex neighborhoods, ensuring linear convergence of gradient-based algorithms. In contrast, within the redundant parameter regime, minima are non-isolated and form lower-dimensional manifolds. These results fundamentally clarify how parameter redundancy shapes optimization dynamics and provide critical theoretical foundations for the efficient training of shallow neural networks.

There exists a significant gap between the theoretical convergence guarantees of deep learning optimization algorithms and their empirical performance, largely due to commonly adopted assumptions—such as Hessian boundedness—that lack empirical validation. Method: We introduce the first trajectory-aware measurement framework tightly aligned with key theoretical quantities, systematically evaluating the validity of mainstream assumptions across diverse architectures and datasets using large-scale training runs. Our framework quantifies dynamic properties—including gradient norms, Hessian spectral characteristics, and loss curvature—along optimization trajectories. Contribution/Results: We find that all examined theoretical assumptions fail to reliably predict actual convergence behavior and exhibit no robust correlation with optimization performance. This work uncovers a fundamental misalignment between theoretical modeling and practice, establishing the first reproducible benchmark for empirically calibrating and reconstructing optimization theory.

This work investigates the approximation capabilities of single-hidden-layer neural networks with globally constrained coefficients—such as those bounded in ℓ¹ norm or exhibiting subexponential growth—when targeting non-analytic functions. Through a deterministic analysis that combines comparison arguments with Bernstein-type estimates, the study demonstrates that such networks remain “rigid” under Gevrey-class activation functions: their approximation error is fundamentally governed by the best polynomial approximation rate, with only exponentially small residual terms. The findings establish that, even when employing non-analytic activations, shallow networks with coefficient constraints cannot surpass classical polynomial approximation rates, thereby revealing an intrinsic limitation in their representational power.

This work investigates the approximation capability of ReLU neural networks with jointly tunable width \(N\) and depth \(L\) for infinitely smooth analytic functions. By carefully constructing networks to approximate power functions, multivariate multiplication, and polynomials, the study establishes, for the first time within a joint \((N, L)\) parameterization framework, an approximation error bound of \(O(N^{-C L^\tau})\) with constants \(C > 0\) and \(\tau > 0\). Notably, when \(N \asymp L^d\), the exponent satisfies \(\tau = 1\), substantially improving upon classical results for finitely smooth functions. This finding underscores the dominant role of depth in approximating analytic functions and reveals a novel scaling relationship between width \(N\) and \(L^d\).

This study addresses the lack of systematic understanding of the loss landscape in physics-informed neural networks (PINNs), particularly under differential operator–based physics losses. For the first time in scientific machine learning, the authors introduce loss landscape analysis by integrating multiple visualization and geometric techniques to systematically compare the optimization landscapes defined by the Deep Ritz method and strong-form residual losses. Their findings reveal that both types of physics-informed losses yield locally smooth, well-conditioned, and nearly convex loss landscapes whose geometric properties closely resemble those of conventional data-driven problems. Notably, these landscapes exhibit favorable structural characteristics near the true solution, challenging the prevailing view that PINN loss landscapes are inherently highly complex. This work thus provides a theoretical foundation for the observed stability and effectiveness of PINN optimization.

This work addresses the inefficiency and weak theoretical guarantees of neural networks in approximating analytic functions and general $L^p$ functions. To overcome these limitations, the authors propose an efficient ReLU network architecture based on a three-dimensional design, which explicitly constructs sawtooth functions to achieve enhanced approximation capabilities. The proposed method significantly improves the exponential approximation rates for a broad class of analytic functions and, for the first time, establishes a high-order, non-asymptotic quantitative approximation theory for general $L^p$ functions. Notably, this approach achieves superior approximation performance while maintaining parameter efficiency and providing rigorous theoretical guarantees.

This work addresses the challenge of characterizing the highly complex loss landscape in large language model (LLM) pretraining, where existing theories struggle to balance analytical tractability with accurate dynamic prediction. By performing Taylor expansions of both the model and loss function at mid-training, the authors construct a local quadratic approximation and combine it with Lanczos quadrature and Hessian spectral estimation. For the first time, they validate this approach on a 150M-parameter LLM trained on 3B tokens, demonstrating predictive accuracy over a training window spanning 10% of total steps. Their analysis reveals that the quadratic model faithfully captures optimization trajectories, that the tail structure of the Hessian spectrum is strongly influenced by batch size, preconditioning, and training stage, and that optimization typically resides in a stochastic edge-of-stability regime dictated by batch size—uncovering a deep connection between local stability and hyperparameter choice.

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of $\varepsilon>0$, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $\mathrm{poly}(\log\log(1/\varepsilon))$ overhead in size, while the converse provably requires $\Omega(\log(1/\varepsilon))$ parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Existing quadratic constraint approaches for characterizing neural network activation functions are overly conservative, limiting the precision of reachability and safety analyses. This work proposes a domain-dependent framework for verifiable quadratic inequalities: it generates candidate constraints via local sampling and employs sum-of-squares (SOS) certificates to ensure global validity, yielding tight and sound quadratic representations for scalar nonlinearities. The method transcends the limitations of conventional sector or slope bounds by incorporating neuron-wise dependencies and local bound refinement—particularly for ReLU networks—to reduce conservatism. It is compatible with convex quadratic programming, semialgebraic set descriptions, and integral quadratic constraint (IQC) techniques. Experiments demonstrate that the framework significantly improves analysis accuracy for smooth activations such as tanh and extends effectively to systems involving saturation-type nonlinearities.

This work systematically investigates the mathematical expressivity of neural networks, with a focus on their approximation efficiency across various function spaces. By integrating tools from functional analysis, approximation theory, and Sobolev space theory, it traces the theoretical development from the universal approximation property of single-hidden-layer networks to modern insights into depth–width trade-offs, parameter efficiency, and the influence of target function smoothness on approximation rates. The study particularly highlights the advantage of deep architectures in achieving superior parameter efficiency for structured function classes. It further incorporates recent models such as Kolmogorov–Arnold Networks (KANs) into this analytical framework, establishing a unified qualitative and quantitative understanding of neural network approximation capabilities and elucidating the pivotal role of depth in enhancing approximation efficiency.