DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

2026-09-28

AI Summary

This study addresses the physical inconsistency of trajectories in monocular video-based hand-object interaction reconstruction by proposing a framework that integrates geometric diffusion with coupled dynamics. Methodologically, geometry-guided diffusion is employed to generate initial motions, while hand-object coupled dynamics supervision based on the Newton-Euler equations is introduced to enforce interaction plausibility through contact force transmission constraints. Furthermore, an active driving penalty mechanism grounded in probabilistic priors is designed to ensure that the reconstructed motions adhere to physical laws. This work significantly improves both the physical consistency and accuracy of hand-object interaction reconstruction, providing reliable support for world model generation and robotic manipulation learning.

Abstract

We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.

Problem

• Hand-object interaction reconstruction
• Monocular RGB video
• Physics-aware modeling
• Mechanical consistency
• Interaction dynamics

Innovation

• Hand-Object Interaction Reconstruction
• Physics-aware Modeling
• Coupled Dynamics
• Diffusion Refinement
• Probabilistic Prior