UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment

2025-06-02

AI Summary

Existing implicit methods for high-fidelity dynamic human avatar reconstruction from multi-view video suffer from geometric detail loss under 4K+ rendering due to depth misalignment and surface drift. To address this, we propose a cascaded 3D point trajectory supervision framework that jointly leverages a 2D video point tracker and an implicit deformation model to enforce multi-level geometric alignment—both at vertex and texture levels. Our end-to-end trainable architecture integrates differentiable template mesh deformation with implicit neural representations. Evaluated on a newly constructed multi-view dataset featuring 40 synchronized cameras, 6K resolution, and long-duration sequences, our method achieves significant improvements in geometric accuracy and 4K+ rendering fidelity over state-of-the-art approaches. The core contribution is the first incorporation of dense 2D point trajectory supervision into implicit human modeling, effectively mitigating inaccurate surface tracking—a persistent challenge in dynamic human reconstruction.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Learning an animatable and clothed human avatar model with vivid dynamics and photorealistic appearance from multi-view videos is an important foundational research problem in computer graphics and vision. Fueled by recent advances in implicit representations, the quality of the animatable avatars has achieved an unprecedented level by attaching the implicit representation to drivable human template meshes. However, they usually fail to preserve the highest level of detail, particularly apparent when the virtual camera is zoomed in and when rendering at 4K resolution and higher. We argue that this limitation stems from inaccurate surface tracking, specifically, depth misalignment and surface drift between character geometry and the ground truth surface, which forces the detailed appearance model to compensate for geometric errors. To address this, we propose a latent deformation model and supervising the 3D deformation of the animatable character using guidance from foundational 2D video point trackers, which offer improved robustness to shading and surface variations, and are less prone to local minima than differentiable rendering. To mitigate the drift over time and lack of 3D awareness of 2D point trackers, we introduce a cascaded training strategy that generates consistent 3D point tracks by anchoring point tracks to the rendered avatar, which ultimately supervises our avatar at the vertex and texel level. To validate the effectiveness of our approach, we introduce a novel dataset comprising five multi-view video sequences, each over 10 minutes in duration, captured using 40 calibrated 6K-resolution cameras, featuring subjects dressed in clothing with challenging texture patterns and wrinkle deformations. Our approach demonstrates significantly improved performance in rendering quality and geometric accuracy over the prior state of the art.

Problem

• Improving detail preservation in high-resolution animatable human avatars
• Addressing surface drift and misalignment in avatar geometry
• Enhancing 3D deformation accuracy using 2D video trackers

Innovation

• Latent deformation model for accurate surface tracking
• 2D video point trackers guide 3D deformation
• Cascaded training strategy for consistent 3D point tracks