approximation theory

This paper addresses the fundamental problem of unclear generalization guarantees in machine learning function approximation—specifically, the lack of theoretical foundations for model performance on unknown manifolds and unseen data. Methodologically, it departs from explicit geometric modeling of manifolds (e.g., via Laplace–Beltrami operators or atlases) and instead integrates classical approximation theory, spectral graph theory, differential geometry, and physics-informed embedding to systematically characterize the approximation mechanisms of diverse paradigms—including deep/shallow neural networks, neural operators, Transformers, and physics-informed neural surrogates. Key contributions include: (i) a precise delineation of expressive capacity boundaries across mainstream architectures; (ii) a robust generalization analysis framework that requires no prior knowledge of manifold geometry; and (iii) the first unifying theoretical perspective for manifold learning and scientific machine learning grounded explicitly in approximation theory.

This work investigates the approximation capability of ReLU neural networks with jointly tunable width \(N\) and depth \(L\) for infinitely smooth analytic functions. By carefully constructing networks to approximate power functions, multivariate multiplication, and polynomials, the study establishes, for the first time within a joint \((N, L)\) parameterization framework, an approximation error bound of \(O(N^{-C L^\tau})\) with constants \(C > 0\) and \(\tau > 0\). Notably, when \(N \asymp L^d\), the exponent satisfies \(\tau = 1\), substantially improving upon classical results for finitely smooth functions. This finding underscores the dominant role of depth in approximating analytic functions and reveals a novel scaling relationship between width \(N\) and \(L^d\).

This work establishes the first rigorous quantitative error bounds on the approximation capacity of Neural Ordinary Differential Equations (Neural ODEs) for the canonical quadratic nonlinear case, addressing the fundamental question of how “depth” and “nonlinearity strength” comparatively govern representational power. Method: Leveraging well-posedness theory for differential equations, flow map analysis, and function-space approximation theory—integrated with continuous-depth modeling and polynomial dynamical systems tools—we derive precise characterizations of expressivity. Contribution/Results: We prove that expressive power stems fundamentally from iterative depth composition rather than per-layer nonlinearity complexity; we provide explicit upper bounds on approximation error, demonstrating that shallow quadratic dynamics can equivalently approximate deep, highly nonlinear models; and we deliver critical theoretical foundations for lightweight continuous-depth learning. Furthermore, our analysis draws conceptual parallels to analog differential analyzers and polynomial differential-algebraic equation (DAE) theory, thereby deepening the mechanistic understanding of depth in neural ODEs.

This work identifies the fundamental limitations of shallow neural networks in approximating and learning high-frequency signals under finite machine precision and computational constraints. Methodologically, it employs matrix condition number analysis, gradient flow modeling, frequency-domain error decomposition, and large-scale numerical experiments. The study establishes, for the first time, quantitative lower bounds on numerical approximation error, characterizes computational complexity bottlenecks, and reveals condition-number-driven instability in high-frequency approximation—thereby unifying accuracy, cost, and stability into a precise theoretical framework. Theoretical analysis and empirical validation jointly demonstrate that: (i) approximation error for high-frequency components is dominated by exponentially deteriorating condition numbers; (ii) the error lower bound grows exponentially with signal frequency; and (iii) computational cost increases polynomially to exponentially with frequency. These results provide critical theoretical criteria and practical design principles for modeling high-frequency signals with neural networks.

Approximating high-dimensional, low-smoothness functions—common in cross-domain learning (e.g., invariant learning, transfer learning, SAR imaging)—remains challenging due to limitations of conventional symmetric or positive-definite kernels. Method: This paper proposes a neural network framework based on asymmetric, irregular kernels, breaking away from traditional kernel symmetry/positivity constraints. It systematically constructs generalized translation networks and rotationally banded function kernels—novel asymmetric kernel architectures—and establishes their approximation theory. It further introduces the ReLU^r activation (with non-integer r > 0) and derives its uniform approximation error bound for Sobolev functions. Contribution/Results: Leveraging asymmetric kernel decomposition and Sobolev space analysis, the framework yields tight approximation error estimates for low-smooth, high-dimensional functions. Empirically, it achieves significantly improved cross-domain generalization accuracy under small-sample and low-regularity conditions.

This work addresses the inefficiency and weak theoretical guarantees of neural networks in approximating analytic functions and general $L^p$ functions. To overcome these limitations, the authors propose an efficient ReLU network architecture based on a three-dimensional design, which explicitly constructs sawtooth functions to achieve enhanced approximation capabilities. The proposed method significantly improves the exponential approximation rates for a broad class of analytic functions and, for the first time, establishes a high-order, non-asymptotic quantitative approximation theory for general $L^p$ functions. Notably, this approach achieves superior approximation performance while maintaining parameter efficiency and providing rigorous theoretical guarantees.

This work systematically investigates the mathematical expressivity of neural networks, with a focus on their approximation efficiency across various function spaces. By integrating tools from functional analysis, approximation theory, and Sobolev space theory, it traces the theoretical development from the universal approximation property of single-hidden-layer networks to modern insights into depth–width trade-offs, parameter efficiency, and the influence of target function smoothness on approximation rates. The study particularly highlights the advantage of deep architectures in achieving superior parameter efficiency for structured function classes. It further incorporates recent models such as Kolmogorov–Arnold Networks (KANs) into this analytical framework, establishing a unified qualitative and quantitative understanding of neural network approximation capabilities and elucidating the pivotal role of depth in enhancing approximation efficiency.

This study addresses the computational bottlenecks in scientific computing arising from the infeasibility of exact algorithms for large-scale problems. Through a systematic evaluation of approximation methods across 118 core algorithmic problems—integrating complexity analysis, taxonomies of approximation algorithms, and historical context—the work presents the first large-scale empirical evidence demonstrating that only approximately 20% of these problems derive substantial benefit from approximation. Notably, one-quarter of exponential-time-hard problems admit polynomial-time approximation schemes, and the adoption of approximation strategies increases the proportion of linear-time solvable problems by 23%. By quantifying the trade-offs between accuracy and efficiency, this research offers theoretical insights to guide the design of AI-driven and high-performance algorithms.

This work proposes an operator surrogate modeling framework for partial differential equations (PDEs) and boundary integral equations (BIEs) defined on domains that are diffeomorphic to a reference shape. By leveraging domain pullback and parametric mappings, geometric variations are encoded as parameters, thereby recasting the problem as a parametric PDE. Building upon this formulation, neural and spectral operator models are constructed to approximate the mapping from shape parameters to solution fields. The study establishes, for the first time, unified approximation error bounds with explicit convergence rates for cross-shape generalization tasks. Through a theoretical analysis combining parametric PDE theory, complex analyticity, and principal component-based shape encoding, the approach is shown to guarantee shape-family-uniform error estimates for both elliptic and parabolic PDEs as well as BIEs.

This work investigates the comparative capabilities of compositional and additive approximation approaches under parameter efficiency constraints. By constructing function classes with specific structural properties and leveraging tools from function approximation theory, parameterized complexity analysis, and neural network encoding limitations, the study provides the first explicit example of functions for which compositional methods provably outperform additive ones. The analysis demonstrates that there exist function classes where compositional approximations achieve a strictly faster error decay rate than any additive method subject to bit-encoding constraints, and that the gap in approximation error between the two paradigms can be made arbitrarily large. These results formally establish the theoretical superiority of compositional approaches in certain regimes.