Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera

2024-12-17

arXiv.org

AI Summary

This work addresses the challenging problem of 4D hand motion reconstruction from monocular video under dynamic camera motion—particularly egocentric settings—where existing methods, relying on weak-perspective models, fail to recover accurate global 3D trajectories due to depth ambiguity, self-occlusion, and trajectory drift. We propose the first globally consistent solution, introducing a multi-stage optimization framework that tightly integrates SLAM, a generative interactive hand prior, and hierarchical initialization. Our approach jointly optimizes hand mesh sequences and globally consistent 3D trajectories via differentiable rendering and multi-objective loss functions. Evaluated on diverse indoor and outdoor real-world datasets, our method significantly outperforms state-of-the-art approaches and establishes the first benchmark for 4D hand reconstruction under dynamic camera motion.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We propose Dyn-HaMR, to the best of our knowledge, the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild. Reconstructing accurate 3D hand meshes from monocular videos is a crucial task for understanding human behaviour, with significant applications in augmented and virtual reality (AR/VR). However, existing methods for monocular hand reconstruction typically rely on a weak perspective camera model, which simulates hand motion within a limited camera frustum. As a result, these approaches struggle to recover the full 3D global trajectory and often produce noisy or incorrect depth estimations, particularly when the video is captured by dynamic or moving cameras, which is common in egocentric scenarios. Our Dyn-HaMR consists of a multi-stage, multi-objective optimization pipeline, that factors in (i) simultaneous localization and mapping (SLAM) to robustly estimate relative camera motion, (ii) an interacting-hand prior for generative infilling and to refine the interaction dynamics, ensuring plausible recovery under (self-)occlusions, and (iii) hierarchical initialization through a combination of state-of-the-art hand tracking methods. Through extensive evaluations on both in-the-wild and indoor datasets, we show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery. This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras. Our project page is at https://dyn-hamr.github.io/.

Problem

• Reconstructing 4D hand motion from dynamic monocular videos
• Overcoming weak perspective limitations in hand motion recovery
• Improving depth estimation for moving camera scenarios

Innovation

• Uses SLAM for dynamic camera motion estimation
• Incorporates interacting-hand prior for occlusion handling
• Combines hierarchical initialization with hand tracking