ControlFace: Harnessing Facial Parametric Control for Face Rigging

2024-12-02

arXiv.org

AI Summary

To address the challenges in face reenactment—namely, heavy reliance on subject-specific image data, necessity of per-subject fine-tuning, and difficulty in simultaneously preserving identity fidelity and enabling precise semantic control—this paper proposes a data-free, tuning-free, high-fidelity 3D Morphable Model (3DMM)-driven method. Our approach introduces a dual-branch U-Net architecture (FaceNet + generation branch), conditioned on 3DMM-rendered features; a Control Mixing Module for disentangled manipulation of multi-attribute parameters (pose, expression, illumination); and a Reference-Control Guidance mechanism to enhance identity consistency. The model is trained end-to-end on video data without requiring per-subject adaptation. Extensive evaluation across multiple benchmarks demonstrates a 12.6% improvement in identity similarity and a 31.4% reduction in control error, while supporting real-time, high-fidelity reenactment with strong generalization across unseen identities and domains.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Manipulation of facial images to meet specific controls such as pose, expression, and lighting, also known as face rigging, is a complex task in computer vision. Existing methods are limited by their reliance on image datasets, which necessitates individual-specific fine-tuning and limits their ability to retain fine-grained identity and semantic details, reducing practical usability. To overcome these limitations, we introduce ControlFace, a novel face rigging method conditioned on 3DMM renderings that enables flexible, high-fidelity control. We employ a dual-branch U-Nets: one, referred to as FaceNet, captures identity and fine details, while the other focuses on generation. To enhance control precision, the control mixer module encodes the correlated features between the target-aligned control and reference-aligned control, and a novel guidance method, reference control guidance, steers the generation process for better control adherence. By training on a facial video dataset, we fully utilize FaceNet's rich representations while ensuring control adherence. Extensive experiments demonstrate ControlFace's superior performance in identity preservation and control precision, highlighting its practicality. Please see the project website: https://cvlab-kaist.github.io/ControlFace/.

Problem

• Enhances facial image manipulation
• Improves control precision in face rigging
• Preserves identity and semantic details

Innovation

• Dual-branch U-Nets
• Control mixer module
• Reference control guidance