When are dynamical systems learned from time series data statistically accurate?

2024-11-09

Neural Information Processing Systems

AI Summary

This work addresses the lack of statistical fidelity in time-series modeling of dynamical systems, where conventional generalization error fails to guarantee accurate reproduction of physical invariants—such as invariant measures and Lyapunov exponents. To this end, we establish, for the first time, an ergodic-theoretic statistical generalization framework, explicitly identifying preservation of the physical measure as the central learning objective. We uncover a fundamental cause of statistical distortion in Neural ODEs applied to dynamical system regression: their neglect of Jacobian structural constraints. We theoretically prove—and empirically validate—that Jacobian regularization substantially enhances statistical generalization. On benchmark chaotic systems (Lorenz, Rössler), incorporating Jacobian information reduces estimation errors in invariant measures, statistical moments, and Lyapunov exponents by an average of 47%, outperforming MLPs, ResNets, Fourier Neural Networks, and RNNs.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Large-scale security measurements

Abstract

Conventional notions of generalization often fail to describe the ability of learned models to capture meaningful information from dynamical data. A neural network that learns complex dynamics with a small test error may still fail to reproduce its emph{physical} behavior, including associated statistical moments and Lyapunov exponents. To address this gap, we propose an ergodic theoretic approach to generalization of complex dynamical models learned from time series data. Our main contribution is to define and analyze generalization of a broad suite of neural representations of classes of ergodic systems, including chaotic systems, in a way that captures emulating underlying invariant, physical measures. Our results provide theoretical justification for why regression methods for generators of dynamical systems (Neural ODEs) fail to generalize, and why their statistical accuracy improves upon adding Jacobian information during training. We verify our results on a number of ergodic chaotic systems and neural network parameterizations, including MLPs, ResNets, Fourier Neural layers, and RNNs.

Problem

• Assessing statistical accuracy of learned dynamical systems from time series
• Conventional generalization fails to capture physical behavior in dynamics
• Improving neural network generalization for ergodic and chaotic systems

Innovation

• Ergodic theoretic approach for generalization
• Neural ODEs with Jacobian information
• Captures invariant physical measures