Navigating Large-Pose Challenge for High-Fidelity Face Reenactment with Video Diffusion Model

2025-07-22

AI Summary

Existing facial reenactment methods suffer from geometric distortions under large pose variations and struggle to simultaneously preserve identity fidelity and video quality. To address this, we propose a high-fidelity reenactment framework built upon video diffusion models. Our core innovation is the Warping Feature Mapper (WFM), which projects warped frames into the dynamic-aware latent space of a pretrained image-video foundation model, enabling distortion correction and inter-frame consistency modeling via large-scale facial motion priors. The method integrates implicit keypoint extraction, motion alignment, and diffusion-based generation to establish a synergistic mechanism between fine-grained motion representation and generative priors. Extensive experiments demonstrate that our approach significantly improves pose reconstruction accuracy, identity preservation, and visual quality—especially under extreme poses—outperforming state-of-the-art methods in both quantitative metrics and qualitative assessment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Face reenactment aims to generate realistic talking head videos by transferring motion from a driving video to a static source image while preserving the source identity. Although existing methods based on either implicit or explicit keypoints have shown promise, they struggle with large pose variations due to warping artifacts or the limitations of coarse facial landmarks. In this paper, we present the Face Reenactment Video Diffusion model (FRVD), a novel framework for high-fidelity face reenactment under large pose changes. Our method first employs a motion extractor to extract implicit facial keypoints from the source and driving images to represent fine-grained motion and to perform motion alignment through a warping module. To address the degradation introduced by warping, we introduce a Warping Feature Mapper (WFM) that maps the warped source image into the motion-aware latent space of a pretrained image-to-video (I2V) model. This latent space encodes rich priors of facial dynamics learned from large-scale video data, enabling effective warping correction and enhancing temporal coherence. Extensive experiments show that FRVD achieves superior performance over existing methods in terms of pose accuracy, identity preservation, and visual quality, especially in challenging scenarios with extreme pose variations.

Problem

• High-fidelity face reenactment under large pose changes
• Addressing warping artifacts in motion transfer
• Enhancing temporal coherence and identity preservation

Innovation

• Uses implicit facial keypoints for motion alignment
• Introduces Warping Feature Mapper for warping correction
• Leverages pretrained I2V model for temporal coherence