Inference-Time Search using Side Information for Diffusion-based Image Reconstruction

2025-10-02

AI Summary

Existing diffusion models for image reconstruction inverse problems often neglect side information, leading to suboptimal reconstructions under severely ill-posed conditions. This work proposes an inference-time search algorithm that dynamically integrates side information during the sampling process, balancing exploration and exploitation while avoiding gradient-guided reward-hacking artifacts. The core contribution is the first explicit incorporation of side information into the search mechanism of diffusion model inference, coupled with a lightweight fusion module designed for seamless integration into mainstream reconstruction pipelines—including deblurring, super-resolution, and inpainting. Extensive experiments demonstrate significant improvements over state-of-the-art methods across diverse inverse problems, particularly in highly ill-conditioned settings. The implementation is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models have emerged as powerful priors for solving inverse problems. However, existing approaches typically overlook side information that could significantly improve reconstruction quality, especially in severely ill-posed settings. In this work, we propose a novel inference-time search algorithm that guides the sampling process using the side information in a manner that balances exploration and exploitation. This enables more accurate and reliable reconstructions, providing an alternative to the gradient-based guidance that is prone to reward-hacking artifacts. Our approach can be seamlessly integrated into a wide range of existing diffusion-based image reconstruction pipelines. Through extensive experiments on a number of inverse problems, such as box inpainting, super-resolution, and various deblurring tasks including motion, Gaussian, nonlinear, and blind deblurring, we show that our approach consistently improves the qualitative and quantitative performance of diffusion-based image reconstruction algorithms. We also show the superior performance of our approach with respect to other baselines, including reward gradient-based guidance algorithms. The code is available at href{https://github.com/mhdfb/sideinfo-search-reconstruction}{this repository}.

Problem

• Improving diffusion-based image reconstruction using side information
• Addressing reward-hacking artifacts in gradient-based guidance methods
• Enhancing performance for inpainting super-resolution and deblurring tasks

Innovation

• Uses side information during inference-time search
• Balances exploration and exploitation in sampling
• Seamlessly integrates into existing diffusion pipelines