Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods

2026-06-05

AI Summary

This work addresses the lack of theoretical understanding regarding the online learning dynamics and generalization mechanisms of high-dimensional controlled nonlinear dynamical systems, such as neural ordinary differential equations (neural ODEs). For the first time, it systematically applies dynamical mean-field theory to analyze online stochastic gradient descent training of neural ODEs. In the high-dimensional limit, the framework rigorously solves the coupled dynamics of training and inference and analytically derives the associated learning curves. This study establishes the first tractable theoretical framework for understanding both the training dynamics and generalization capabilities of deep continuous models, revealing the precise evolution laws governing high-dimensional neural ODEs under online learning.

Abstract

Neural ordinary differential equations (neural ODEs) have rapidly gained prominence as a powerful and unifying framework for conceptualizing artificial neural networks, elegantly connecting the continuous-time modeling of dynamical systems with the discrete, data-driven paradigm of modern deep learning. Beyond their practical advantages they offer fresh theoretical insights into the training and generalization properties of neural networks. The distinctive feature of this framework is its dual dynamical nature: inference dynamics, which govern the ODE evolution during forward computation, and training dynamics, which control the optimization of model parameters. This makes neural ODEs a particularly well-suited theoretical framework for studying a large variety of settings such as multi-layer neural networks (ResNets for example), autoregressive models (with next-token generation dynamics), generative models, and recurrent neural networks in theoretical neuroscience. In this work, we introduce a theoretically grounded class of models for studying neural ODEs trained via online stochastic gradient descent. We solve the training dynamics of these models via dynamical mean field theory and derive learning curves in the high-dimensional limit.

Problem

• neural ODEs
• training dynamics
• high-dimensional systems
• online stochastic gradient descent
• learning curves

Innovation

• neural ODEs
• dynamical mean field theory
• online stochastic gradient descent
• high-dimensional limit
• learning curves