motion prior learning

Generating realistic and physically plausible 3D human-object interaction (HOI) motions remains challenging, primarily because natural language often fails to fully capture complex physical constraints. This work proposes ViHOI, a novel framework that leverages visual and textual priors from 2D images to guide 3D HOI synthesis—marking the first approach to do so. Specifically, ViHOI extracts multimodal priors using large vision-language models, compresses these features via a Q-Former adapter, and injects them into a diffusion-based generative model through a layer-decoupled strategy. Extensive experiments demonstrate that ViHOI significantly outperforms existing methods across multiple benchmarks and exhibits strong generalization capabilities on unseen objects and interaction categories.

To address the violation of conservation laws and prediction instability in data-driven dynamical system modeling—caused by the absence of physical priors—this paper proposes the first systematic framework for integrating multiple physics-based constraints into diffusion models. Our method embeds distributional symmetries (e.g., rotational/translational invariance), PDE-based dynamics, and energy/momentum conservation directly into the generative process via symmetry-aware encoding, physics-informed projection, PDE-guided score matching, and explicit conservation-law regularization. Evaluated on rigid-body motion and fluid flow systems, our approach yields trajectories and velocity fields with high fidelity, long-term stability, and verifiably consistent physical behavior. It significantly improves prediction accuracy and out-of-distribution generalization robustness. This work establishes the first diffusion-based generative paradigm for physics-informed AI that is both theoretically interpretable—grounded in rigorous physical principles—and empirically verifiable through quantitative conservation metrics.

Existing video generation methods often suffer from a lack of physical consistency, manifesting as object drift, implausible collisions, and unrealistic material responses. This work proposes a controllable video generation framework that achieves physically plausible synthesis without relying on simulators or geometric reconstruction during inference. By leveraging a large-scale dataset of physics-simulated videos, the approach combines ControlNet fine-tuning conditioned on pixel-aligned physical attribute maps with differentiable reward optimization guided by a vision-language model (VLM), enabling continuous, interpretable, and precise control over physical properties such as friction and elasticity. Integrating physics-supervised fine-tuning with VLM-based feedback for the first time, the method substantially outperforms strong baselines on the Physics-IQ benchmark, and human evaluations confirm its superior physical realism and controllability in generated videos.

This work addresses ultra-low-bitrate compression of dynamically varying scene videos. Instead of conventional content-based modeling, it leverages natural motion patterns—such as flower swaying or boat drifting—as priors. Methodologically, it introduces, for the first time, a lightweight generative prior for scene motion, establishing a novel framework comprising dense motion representation, sparse motion coding, and optical-flow-guided diffusion decoding—fully abandoning inter-frame prediction. Key contributions are: (1) learning compact, generalizable motion priors from common dynamic scenes; and (2) designing a flow-driven generative decoding mechanism enabling high-fidelity dynamic reconstruction. Experiments demonstrate substantial gains over VVC across diverse dynamic sequences, maintaining strong motion consistency and visual quality at ultra-low bitrates (0.01–0.1 bpp), with comprehensive improvements in rate-distortion performance.

To address motion discontinuity and poor temporal consistency in keyframe-based video interpolation, this paper proposes a lightweight bidirectional diffusion sampling framework. Without retraining large-scale models, it fine-tunes pre-trained image-to-video diffusion models (e.g., Sora-like architectures) to enable bidirectional temporal modeling. The method initiates collaborative sampling from both end keyframes and introduces an overlapping estimation fusion strategy to enhance motion plausibility and structural fidelity of intermediate frames. To our knowledge, this is the first work to efficiently adapt unidirectional image-to-video diffusion models for keyframe interpolation. Extensive experiments demonstrate that our approach significantly outperforms optical-flow-based methods and existing diffusion-based interpolation techniques across multiple benchmarks, achieving state-of-the-art performance in visual quality, motion smoothness, and temporal consistency.

Existing approaches struggle to efficiently model the multimodal distribution of future scene dynamics from partial observations. To address this challenge, this work proposes GARFIELD, a novel framework that explicitly constructs a structured spatiotemporal latent variable distribution to represent all plausible future motions conditioned on an input image and sparse constraints. By integrating object-aware latent space modeling, a deterministic density decoder, and probabilistic trajectory generation, GARFIELD enables joint sampling, local uncertainty estimation, progressive constraint fusion, and interactive density queries—without requiring large-scale video generation or Monte Carlo sampling. Experiments demonstrate that GARFIELD achieves trajectory sampling 97× faster than large video generation models and density estimation two orders of magnitude faster, while delivering competitive performance in motion planning tasks and supporting real-time interaction.

This work addresses the "motion entanglement" problem in existing video diffusion models, where independent motion sources—such as camera and object movements—are erroneously coupled, leading to physically implausible videos. To mitigate this, the authors propose a Self-Imagined Fine-Tuning (SIFT) paradigm that eschews direct reconstruction of real videos and instead trains the model on its own generated samples. SIFT integrates motion-aware discriminative supervision with a progressive hard example replay strategy, leveraging free-form text prompts to encompass rare or finely disentangled motion scenarios and thereby alleviating data bias. Experimental results demonstrate that SIFT substantially enhances the physical plausibility, motion disentanglement, and controllability of generated videos, outperforming current methods across multiple quantitative metrics and visual evaluations.

This work addresses the challenge of enabling humanoid robots to achieve both human-like naturalness and robust locomotion over complex terrains. To this end, the authors propose Terrain-conditioned Generative Motion Prior (T-GMP), a novel approach that integrates terrain information directly into motion prior learning. T-GMP employs a conditional variational autoencoder to model terrain-dependent latent motion manifolds from a limited set of expert demonstrations. By incorporating adversarial training, a foot-placement penalty mechanism, and a terrain-aware discriminator, the method enables a unified policy to generate adaptive and diverse human-like gaits. Experimental results demonstrate that T-GMP outperforms existing methods in terms of traversal success rate and motion smoothness while preserving strong biomechanical naturalness and physical coordination.

Existing Hamiltonian video dynamics models struggle to generalize to unseen time steps in non-conservative, externally driven scenarios, leading to failure in multi-scale prediction. This work identifies two distinct mechanisms underlying their poor temporal generalization in continuous-time modeling and proposes targeted improvements: incorporating a constrained action-force mapping and a higher-order numerical integrator within a framework that combines Hamiltonian generative networks with continuous-time energy function modeling. The resulting approach substantially enhances prediction stability and accuracy at time resolutions outside the training distribution, enabling reliable long-horizon video dynamics forecasting—far beyond the temporal scales seen during training—across diverse dissipative and externally forced environments.

This work addresses the challenge of integrating data-driven deep learning with model-based planning to enable flexible, efficient, and realistic robot motion planning. It proposes a generative framework that, for the first time, compresses high-dimensional continuous trajectories into a hierarchical discrete latent space using a high-ratio autoencoder, and directly optimizes arbitrary task-specific objective functions in this latent space at test time to search for feasible motion plans. The approach requires no task-specific training and naturally supports multi-agent scenario synthesis and closed-loop planning. Experiments on the nuPlan and Waymo Open Motion Dataset demonstrate state-of-the-art performance in both closed-loop motion planning and multi-agent guided generation tasks.