Feature-Guided Diffusion for Non-Differentiable Inverse Rendering

2026-07-19

AI Summary

This work proposes Feature-Informed Diffusion Evolution (FIDE), a novel framework that introduces dense visual features into black-box inverse rendering. Traditional inverse rendering methods rely on differentiable renderers and gradient-based optimization, which are prone to local minima and require careful initialization, while derivative-free approaches remain sensitive to initial conditions. FIDE circumvents these limitations by leveraging a Vision Transformer to extract image features that guide a diffusion model in generating high-quality parameter candidates, followed by closed-loop refinement via CMA-ES. Notably, the method operates without gradients or task-specific initialization and demonstrates significantly accelerated convergence across diverse tasks—including path tracing, vector spline fitting, Voronoi shader inversion, and robot control—effectively overcoming stagnation issues commonly observed in gradient-based approaches.

Abstract

Inverse rendering is traditionally solved via differentiable renderers and gradient descent, which requires substantial problem-specific engineering and is prone to getting stuck in local minima due to ambiguities. Derivative-free approaches alleviate engineering requirements, but often heavily depend on a good problem initialization. In this work, we propose Feature-Informed Diffusion Evolution (FIDE), a fully black-box framework that requires no gradients or specific initialization: the renderer is treated as an opaque function whose only requirement is to produce images. Our key insight is feature guiding: rather than reducing each candidate rendering to a scalar loss value, we use a Vision Transformer (ViT) to extract dense visual features from it. We subsequently use these features to train a diffusion-based candidate proposal model, allowing the network to use visual cues to predict parameters that would match the target image. The candidate solutions proposed by this diffusion model are then refined in a closed loop with a CMA evolution strategy, continuously narrowing the proposal region as optimization progresses. We validate across diverse inverse problems from path tracing, vector splines, Voronoi shaders, and robotics, and demonstrate that feature-guiding substantially improves convergence over scalar-loss baselines and reliably escapes local minima where gradient-based methods stall.

Problem

• inverse rendering
• non-differentiable
• black-box optimization
• local minima
• parameter recovery

Innovation

• feature-guided diffusion
• inverse rendering
• black-box optimization
• Vision Transformer
• CMA-ES