Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case

2026-08-06

AI Summary

This work addresses the tendency of existing commercial generative editors to produce physically implausible results in shadow removal by erroneously treating shadows as intrinsic material or structural attributes, often due to neglecting the physical principles of illumination. To mitigate this, the authors propose an agent-based iterative editing framework that integrates physical priors of shadow formation—specifically, that shadows arise from occlusion of light sources. The framework employs a generator to construct guided probes and an evaluator that leverages these physical priors to detect inconsistencies and trigger retries. By sampling multiple candidates and selecting the output with highest physical consistency, the method explicitly embeds shadow physics into both prompt engineering and evaluation. Evaluated on the ShadowRemovalRefine benchmark, it achieves a CDD score of 0.0075, a 47% improvement over the current state of the art, significantly enhancing both shadow removal quality and scene fidelity.

Abstract

Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems. This raises a natural question: do they reduce the need for classic, physics-informed low-level vision? We study this through shadow removal, a problem shaped by scene geometry, illumination, materials, and occluders, where paired shadow and shadow-free data are hard to collect at scale. We find that a commercial generative editor, used directly, can produce clean shadow-free edits that preserve surface texture and local appearance. However, this comes with a new failure mode: the same editor can regenerate scene content, hallucinate objects, or misread a shadow as material or geometry, producing plausible but physically wrong edits. We address this with an agentic candidate-selection pipeline: the editor generates a guided probe, an evaluator screens for major failures, retries when needed, samples multiple candidates, filters them, and selects a final result balancing shadow removal against scene preservation. Grounding this process in shadow-formation physics makes it more reliable: prompting the generator and evaluator to treat shadows as illumination effects caused by light occlusion, not material or object structure, measurably improves quality and consistency. On the ShadowRemovalRefine benchmark, our physics-oriented pipeline achieves a CDD of 0.0075, reducing CDD by at least 47% over the strongest prior method. These results suggest that commercial vision-language models do not replace classic low-level vision priors; instead, such priors remain useful for constraining and steering physically underconstrained generation.

Problem

• shadow removal
• vision-language models
• physical consistency
• image editing
• hallucination

Innovation

• shadow removal
• vision-language models
• physics-informed priors
• agentic editing
• candidate selection