Turn That Frown Upside Down: FaceID Customization via Cross-Training Data

2025-01-26

AI Summary

Existing face ID customization methods merely replicate input faces, lacking controllable diversity in expression, pose, and viewpoint—severely limiting personalized expressiveness. To address this, we propose a novel cross-variant face customization paradigm. We introduce CrossFaceID, the first large-scale public dataset comprising 40K text–image pairs from 2,000 identities, enabling fine-grained identity-preserving editing. Our method integrates cross-sample supervised fine-tuning, text–image alignment modeling, FaceID feature disentanglement training, and controllable prompt engineering. Experiments demonstrate that our approach achieves 98.7% identity fidelity while improving facial attribute editing accuracy by 42.3% over baselines. It generalizes effectively to unseen expression–pose combinations and supports zero-shot style transfer. All code, data, and models are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Large-scale security measurements

Abstract

Existing face identity (FaceID) customization methods perform well but are limited to generating identical faces as the input, while in real-world applications, users often desire images of the same person but with variations, such as different expressions (e.g., smiling, angry) or angles (e.g., side profile). This limitation arises from the lack of datasets with controlled input-output facial variations, restricting models' ability to learn effective modifications. To address this issue, we propose CrossFaceID, the first large-scale, high-quality, and publicly available dataset specifically designed to improve the facial modification capabilities of FaceID customization models. Specifically, CrossFaceID consists of 40,000 text-image pairs from approximately 2,000 persons, with each person represented by around 20 images showcasing diverse facial attributes such as poses, expressions, angles, and adornments. During the training stage, a specific face of a person is used as input, and the FaceID customization model is forced to generate another image of the same person but with altered facial features. This allows the FaceID customization model to acquire the ability to personalize and modify known facial features during the inference stage. Experiments show that models fine-tuned on the CrossFaceID dataset retain its performance in preserving FaceID fidelity while significantly improving its face customization capabilities. To facilitate further advancements in the FaceID customization field, our code, constructed datasets, and trained models are fully available to the public.

Problem

• Facial Expression Diversity
• Perspective Variation
• Personalized Facial Recognition

Innovation

• CrossFaceID
• Expression Diversity
• Viewpoint Variation