ViDS: Video Diffusion Shader using 3D Face Tracking

2026-07-27

AI Summary

This work proposes a high-fidelity portrait video animation method that preserves the identity and appearance of a reference image while enabling rich facial expressions and precise pose control. The approach leverages a 3D Morphable Model (3DMM) to reconstruct an identity-specific mesh, incorporates expression and pose parameters extracted from a driving video, and employs 3D normal maps to provide geometric guidance. A video diffusion model acts as a neural shader to synthesize photorealistic outputs. The method innovatively integrates accurate geometric constraints derived from 3DMM tracking and introduces an autoregressive diffusion sampling strategy, effectively supporting long-duration video generation and mitigating inter-frame discontinuities. Experiments demonstrate that the proposed approach significantly outperforms existing methods—particularly those relying on keypoints or implicit motion representations—in terms of identity preservation, expression detail, and pose consistency, with ablation studies confirming the contribution of each component.

Abstract

We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/

Problem

• portrait animation
• identity preservation
• expression control
• 3D face tracking
• video diffusion

Innovation

• Video Diffusion
• 3D Face Tracking
• Neural Shader
• 3DMM
• Autoregressive Sampling