🤖 AI Summary
This work addresses the challenges of stability degradation and accuracy loss in neural differential equations during long-term integration, as well as the high computational cost associated with training on long trajectories. To overcome these issues, the authors propose two low-cost Jacobian regularization strategies: one that directly computes directional derivatives when the dynamics are known, and another that employs finite-difference approximations when they are not. These approaches significantly enhance the long-term simulation stability of models trained on short trajectories while substantially reducing training overhead. The method demonstrates robustness and scalability across multiple ordinary and partial differential equation systems, offering an efficient pathway for learning large-scale dynamical systems.
📝 Abstract
Hybrid models and Neural Differential Equations (NDE) are getting increasingly important for the modeling of physical systems, however they often encounter stability and accuracy issues during long-term integration. Training on unrolled trajectories is known to limit these divergences but quickly becomes too expensive due to the need for computing gradients over an iterative process. In this paper, we demonstrate that regularizing the Jacobian of the NDE model via its directional derivatives during training stabilizes long-term integration in the challenging context of short training rollouts. We design two regularizations, one for the case of known dynamics where we can directly derive the directional derivatives of the dynamic and one for the case of unknown dynamics where they are approximated using finite differences. Both methods, while having a far lower cost compared to long rollouts during training, are successful in improving the stability of long-term simulations for several ordinary and partial differential equations, opening up the door to training NDE methods for long-term integration of large scale systems.