LiveLight: Real-time Streaming Video Relighting with Interactive Control

2026-08-03

AI Summary

This work addresses three key challenges in real-time streaming video relighting with interactive 3D lighting control: dynamic illumination injection, high-quality generation under low computational cost, and temporal coherence. To this end, the authors propose a lightweight diffusion-based framework that incorporates depth-aware multi-plane light irradiance (MPLI) conditioning for dynamic lighting, employs a geometry-guided feedback branch to enhance rendering quality at low NFE (number of function evaluations), and introduces a progressive latent sliding window mechanism to ensure long-sequence temporal consistency. The method achieves real-time video relighting with interactive 3D lighting control for the first time, demonstrating state-of-the-art performance on both real-world and synthetic datasets. It significantly outperforms offline baselines while excelling in runtime speed, temporal stability, lighting controllability, and user preference.

Abstract

We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.

Problem

• video relighting
• real-time streaming
• interactive control
• diffusion model
• 3D lighting

Innovation

• diffusion-based relighting
• real-time video processing
• interactive 3D lighting
• geometry-guided feedback
• streaming video relighting