🤖 AI Summary
Existing Hamiltonian video dynamics models struggle to generalize to unseen time steps in non-conservative, externally driven scenarios, leading to failure in multi-scale prediction. This work identifies two distinct mechanisms underlying their poor temporal generalization in continuous-time modeling and proposes targeted improvements: incorporating a constrained action-force mapping and a higher-order numerical integrator within a framework that combines Hamiltonian generative networks with continuous-time energy function modeling. The resulting approach substantially enhances prediction stability and accuracy at time resolutions outside the training distribution, enabling reliable long-horizon video dynamics forecasting—far beyond the temporal scales seen during training—across diverse dissipative and externally forced environments.
📝 Abstract
World models are typically trained to predict discrete-time physical dynamics with a fixed step size baked into the model weights, preventing prediction at variable temporal resolutions. This matters for hierarchical planning, sim-to-real transfer, and scientific or game-engine applications that must query the same dynamics at multiple timescales. Hamiltonian Generative Networks (HGN) offer a principled path forward, grounding predictions in a continuous-time energy function that is, in principle, independent of the observation frame rate. In practice, however, their temporal generalization breaks down in non-conservative settings. We show that in externally forced, dissipative environments, HGN rollouts at step sizes beyond the training regime fail due to distinct failure modes, including latent magnitude growth driven by an unconstrained action-force map, and global truncation error accumulation from an under-resolved integrator. We identify a targeted fix for each mechanism and demonstrate stable dynamics prediction at temporal resolutions well outside the training distribution. In a detailed analysis, we recommend several strategies for enabling temporal generalization in continuous-time video generation.