Multi-View Face and Gesture Animation with Dynamic Gaussians

2026-08-05

AI Summary

Existing methods struggle to simultaneously achieve high-fidelity facial expressions and detailed hand poses in upper-body virtual human animation, often omitting subtle dynamics crucial for natural interpersonal interaction. This work proposes MVFGA, a novel approach that first models the face and hands separately and then integrates them into a unified parametric upper-body mesh. Coupled with dynamic 3D Gaussian splatting, the method enables multi-view consistent rendering while preserving fine-grained details. Experimental results demonstrate that MVFGA significantly enhances the realism and expressiveness of both facial and hand animations, outperforming current baselines in terms of multi-view rendering quality and motion naturalness.

Abstract

Creating photorealistic 3D human avatars with realistic upper-body motion remains challenging. Existing approaches either focus on the head and overlook hand gestures, or reconstruct the full body but fail to preserve fine-grained facial fidelity and hand pose accuracy. As a result, current methods struggle to capture the subtle dynamics of facial expressions and hand gestures that are crucial for natural human communication. While methods based on full-body parametric models enable avatar reconstruction from monocular or multi-view inputs, they often lack accurate facial animation and detailed hand articulation. To address these limitations, we propose MVFGA, a novel multi-view-consistent pipeline for generating realistic upper-body avatars. Our approach models the face and hands separately and fuses them with a parametric upper-body mesh model, enabling the capture of fine-grained facial expressions and hand poses for accurate upper-body avatar reconstruction. We then splat 3D Gaussians onto the obtained mesh, enabling high-quality rendering of dynamic avatars from novel viewpoints. Furthermore, we introduce MVFGA-MoCap, a multi-view upper-body motion capture dataset featuring controlled facial expression sequences, diverse hand gestures, and free-form communication. Experiments show that MVFGA generates visually realistic avatars with high-fidelity facial expressions and hand motions, outperforming baselines for upper-body avatar animation. Project page: https://dfki-av.github.io/MVFGA/

Problem

• photorealistic avatars
• facial animation
• hand gestures
• upper-body motion
• multi-view consistency

Innovation

• Dynamic Gaussians
• multi-view consistency
• facial animation
• hand gesture modeling
• upper-body avatar