Identity Preserving 3D Head Stylization with Multiview Score Distillation

2024-11-20

arXiv.org

AI Summary

Existing 3D head stylization methods are largely constrained to near-frontal views and suffer from low identity fidelity, failing to meet the demand for omnidirectional artistic rendering in gaming and VR applications. To address this, we propose the first high-fidelity 360° avatar stylization framework. Our method introduces a negative log-likelihood distillation mechanism to enable identity-aware knowledge transfer from diffusion models to PanoHead GAN. We further design a joint optimization scheme combining multi-view mesh scores and mirrored gradients, and propose a novel score-ranking weighting strategy to enhance view consistency. Extensive qualitative and quantitative evaluations demonstrate state-of-the-art performance: identity similarity improves significantly (+12.7% LPIPS-ID), and the framework enables fully consistent, highly personalized artistic generation across all viewing angles.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

3D head stylization transforms realistic facial features into artistic representations, enhancing user engagement across gaming and virtual reality applications. While 3D-aware generators have made significant advancements, many 3D stylization methods primarily provide near-frontal views and struggle to preserve the unique identities of original subjects, often resulting in outputs that lack diversity and individuality. This paper addresses these challenges by leveraging the PanoHead model, synthesizing images from a comprehensive 360-degree perspective. We propose a novel framework that employs negative log-likelihood distillation (LD) to enhance identity preservation and improve stylization quality. By integrating multi-view grid score and mirror gradients within the 3D GAN architecture and introducing a score rank weighing technique, our approach achieves substantial qualitative and quantitative improvements. Our findings not only advance the state of 3D head stylization but also provide valuable insights into effective distillation processes between diffusion models and GANs, focusing on the critical issue of identity preservation. Please visit the https://three-bee.github.io/head_stylization for more visuals.

Problem

• Preserve unique identities in 3D head stylization
• Improve stylization quality from 360-degree views
• Enhance identity preservation using diffusion-GAN distillation

Innovation

• Uses PanoHead for 360-degree view synthesis
• Employs negative log-likelihood distillation for identity
• Integrates multi-view grid score and mirror gradients