SpotLight: Shadow-Guided Object Relighting via Diffusion

2026-04-11

AI Summary

Existing diffusion-based neural rendering methods lack explicit, fine-grained illumination control, hindering photorealistic compositing of virtual objects with scene-consistent lighting. This paper proposes a training-free, shadow-guided relighting method: given only a single coarse user-provided shadow mask, it enables precise control over light direction and shadow generation for virtual objects within a pre-trained diffusion model. Our approach integrates physics-inspired shadow priors with generative priors via shadow-mask injection and conditional fine-tuning—demonstrating for the first time that a single coarse shadow suffices for high-fidelity, controllable neural relighting. Quantitative evaluation and user studies show significant improvements over state-of-the-art methods. The framework supports diverse applications, including hand-drawn shadow inputs and full-image relighting, substantially enhancing compositional realism and lighting consistency.

Technology Category

Computer Vision: Diffusion Models for VisionHumans and AI: Game Design — Virtual Humans, NPCs and Autonomous CharactersIntelligent Robots: Manipulation

Application Category

User Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingEconomics, Online Markets and Human Computation: LLM based quality controls for crowd workResponsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent work has shown that diffusion models can serve as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. However, unlike typical physics-based renderers, these neural rendering engines are limited by the lack of manual control over the lighting, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise lighting control can be achieved for object relighting simply by providing a coarse shadow of the object. Indeed, we show that injecting only the desired shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, leverages existing neural rendering approaches and achieves controllable relighting results with no additional training. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting. We also demonstrate other applications, such as hand-scribbling shadows and full-image relighting, demonstrating its versatility.

Problem

• Achieving precise lighting control in neural rendering
• Relighting objects using coarse shadows in diffusion models
• Harmonizing objects and shadows in target images

Innovation

• Uses shadow guidance for object relighting
• Leverages pre-trained diffusion-based neural renderer
• Achieves controllable relighting without additional training