Improving Facial Rig Semantics for Tracking and Retargeting

2025-08-11

AI Summary

Cross-framework facial retargeting—e.g., between 3DMM, FLAME, and MetaHuman—suffers from semantic inconsistency, leading to inaccurate expression recognition and animation control. To address this, we propose a semantics-aware facial retargeting method. Our approach features: (1) a Simon-Says–based expression calibration mechanism that enables cross-framework semantic alignment; (2) volumetric deformation coupled with implicit differentiation to fine-tune tracker outputs within the target rig space—without modifying the black-box tracker; and (3) a unified parameter mapping framework that models inter-rig correspondences consistently. Experiments demonstrate substantial improvements in geometric fidelity and semantic plausibility for both human-to-human and human-to-virtual-character retargeting. The method exhibits strong robustness and generalization across diverse mainstream facial rigs, maintaining high accuracy without requiring rig-specific adaptations or ground-truth supervision.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this paper, we consider retargeting a tracked facial performance to either another person or to a virtual character in a game or virtual reality (VR) environment. We remove the difficulties associated with identifying and retargeting the semantics of one rig framework to another by utilizing the same framework (3DMM, FLAME, MetaHuman, etc.) for both subjects. Although this does not constrain the choice of framework when retargeting from one person to another, it does force the tracker to use the game/VR character rig when retargeting to a game/VR character. We utilize volumetric morphing in order to fit facial rigs to both performers and targets; in addition, a carefully chosen set of Simon-Says expressions is used to calibrate each rig to the motion signatures of the relevant performer or target. Although a uniform set of Simon-Says expressions can likely be used for all person to person retargeting, we argue that person to game/VR character retargeting benefits from Simon-Says expressions that capture the distinct motion signature of the game/VR character rig. The Simon-Says calibrated rigs tend to produce the desired expressions when exercising animation controls (as expected). Unfortunately, these well-calibrated rigs still lead to undesirable controls when tracking a performance (a well-behaved function can have an arbitrarily ill-conditioned inverse), even though they typically produce acceptable geometry reconstructions. Thus, we propose a fine-tuning approach that modifies the rig used by the tracker in order to promote the output of more semantically meaningful animation controls, facilitating high efficacy retargeting. In order to better address real-world scenarios, the fine-tuning relies on implicit differentiation so that the tracker can be treated as a (potentially non-differentiable) black box.

Problem

• Retargeting facial performance across different rig frameworks
• Calibrating rigs using Simon-Says expressions for accurate motion
• Improving animation control semantics for better tracking results

Innovation

• Utilizes same rig framework for retargeting
• Employs volumetric morphing for rig fitting
• Fine-tunes tracker rig via implicit differentiation