Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration

2024-03-05

arXiv.org

AI Summary

This work addresses the problem of 3D facial mesh registration from a single textureless face image, where conventional geometry-only alignment fails to ensure consistent texture parameterization under expression variations, hindering the construction of topology-consistent head models. To this end, we propose the first geometric-photometric joint alignment framework: it decouples rigid and non-rigid deformations, incorporates differentiable neural rendering constraints, and integrates implicit shape priors, multi-scale photometric loss, and gradient-guided deformation regularization—significantly enhancing robustness to occlusions and low-light conditions. Evaluated on NOVA and CelebA-Mesh benchmarks, our method achieves state-of-the-art performance, reducing vertex error by 21.3%. Notably, it is the first to enable sub-millimeter reconstruction fidelity for fine-grained skin folds and expression details.

Technology Category

Computer Vision: 3D Computer VisionIntelligent Robots: Multimodal Perception & Sensor FusionMachine Learning: Multimodal Learning

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsResponsible Web: Human-perceived consequences of algorithmic deployment on the webSecurity and Privacy: Large-scale security measurements

Problem

• Aligns facial expressions with pixel-level accuracy
• Ensures consistent texture across different expressions
• Combines geometric and photometric alignment automatically

Innovation

• Combines geometric and photometric alignment
• Uses differentiable rendering for joint alignment
• Multiscale optimization ensures robust convergence