High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model

2025-02-27

AI Summary

Existing portrait animation methods struggle with relighting due to their inability to disentangle intrinsic identity/appearance features from extrinsic pose, illumination, and expression attributes. This paper introduces a single-image-driven portrait animation framework built upon a pretrained image-to-video diffusion model. It is the first to explicitly decouple and controllably edit illumination, pose, and expression within video diffusion. We propose a dual-path adapter mechanism—comprising a reference adapter for identity preservation and a shadow adapter for fine-grained illumination control—integrated with 3D mesh reconstruction, illumination-aware rendering prompts, and multi-subspace feature co-modeling. Experiments demonstrate that our method surpasses state-of-the-art approaches across illumination fidelity, visual quality, and temporal coherence. It significantly enhances realism and controllability in relightable portrait animation, establishing a new benchmark.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Ad search and search for Web retail
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Relightable portrait animation aims to animate a static reference portrait to match the head movements and expressions of a driving video while adapting to user-specified or reference lighting conditions. Existing portrait animation methods fail to achieve relightable portraits because they do not separate and manipulate intrinsic (identity and appearance) and extrinsic (pose and lighting) features. In this paper, we present a Lighting Controllable Video Diffusion model (LCVD) for high-fidelity, relightable portrait animation. We address this limitation by distinguishing these feature types through dedicated subspaces within the feature space of a pre-trained image-to-video diffusion model. Specifically, we employ the 3D mesh, pose, and lighting-rendered shading hints of the portrait to represent the extrinsic attributes, while the reference represents the intrinsic attributes. In the training phase, we employ a reference adapter to map the reference into the intrinsic feature subspace and a shading adapter to map the shading hints into the extrinsic feature subspace. By merging features from these subspaces, the model achieves nuanced control over lighting, pose, and expression in generated animations. Extensive evaluations show that LCVD outperforms state-of-the-art methods in lighting realism, image quality, and video consistency, setting a new benchmark in relightable portrait animation.

Problem

• Relightable portrait animation with lighting control
• Separation of intrinsic and extrinsic features
• High-fidelity video diffusion model for realistic animations

Innovation

• Lighting Controllable Video Diffusion
• Separate intrinsic and extrinsic features
• High-fidelity relightable portrait animation