RelightVid: Temporal-Consistent Diffusion Model for Video Relighting

2025-01-27

AI Summary

This work addresses key challenges in video relighting—namely, the scarcity of high-quality training data, the difficulty of jointly ensuring inter-frame coherence and photorealism, and the necessity of preserving original albedo and lighting characteristics. We propose the first end-to-end diffusion-based video relighting framework supporting arbitrary conditioning inputs (background video, text prompts, or environment maps), without requiring explicit geometric or material decomposition. Our method integrates illumination-aware training, dynamic consistency regularization, rendering-guided illumination augmentation, implicit illumination prior transfer, and motion-aware denoising scheduling to enforce albedo constancy and temporal stability. Extensive experiments demonstrate state-of-the-art performance on both synthetic and real-world videos: our approach achieves a 37% reduction in temporal Fréchet Inception Distance (tFID) and a 52% improvement in optical flow consistency over prior methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Diffusion models have demonstrated remarkable success in image generation and editing, with recent advancements enabling albedo-preserving image relighting. However, applying these models to video relighting remains challenging due to the lack of paired video relighting datasets and the high demands for output fidelity and temporal consistency, further complicated by the inherent randomness of diffusion models. To address these challenges, we introduce RelightVid, a flexible framework for video relighting that can accept background video, text prompts, or environment maps as relighting conditions. Trained on in-the-wild videos with carefully designed illumination augmentations and rendered videos under extreme dynamic lighting, RelightVid achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition while preserving the illumination priors of its image backbone.

Problem

• Video Relighting
• Coherence Preservation
• Color Retention

Innovation

• RelightVid
• Lighting Modification
• Video Coherence