🤖 AI Summary
This work establishes the first rigorous quantitative error bounds on the approximation capacity of Neural Ordinary Differential Equations (Neural ODEs) for the canonical quadratic nonlinear case, addressing the fundamental question of how “depth” and “nonlinearity strength” comparatively govern representational power.
Method: Leveraging well-posedness theory for differential equations, flow map analysis, and function-space approximation theory—integrated with continuous-depth modeling and polynomial dynamical systems tools—we derive precise characterizations of expressivity.
Contribution/Results: We prove that expressive power stems fundamentally from iterative depth composition rather than per-layer nonlinearity complexity; we provide explicit upper bounds on approximation error, demonstrating that shallow quadratic dynamics can equivalently approximate deep, highly nonlinear models; and we deliver critical theoretical foundations for lightweight continuous-depth learning. Furthermore, our analysis draws conceptual parallels to analog differential analyzers and polynomial differential-algebraic equation (DAE) theory, thereby deepening the mechanistic understanding of depth in neural ODEs.
📝 Abstract
This work focuses on deriving quantitative approximation error bounds for neural ordinary differential equations having at most quadratic nonlinearities in the dynamics. The simple dynamics of this model form demonstrates how expressivity can be derived primarily from iteratively composing many basic elementary operations, versus from the complexity of those elementary operations themselves. Like the analog differential analyzer and universal polynomial DAEs, the expressivity is derived instead primarily from the"depth"of the model. These results contribute to our understanding of what depth specifically imparts to the capabilities of deep learning architectures.