Diffusion State-Guided Projected Gradient for Inverse Problems

2024-10-04

arXiv.org

AI Summary

Existing diffusion-based inverse solvers suffer from reconstruction artifacts—particularly in image inpainting—due to inaccurate approximation of the measurement gradient, which causes the generated samples to deviate from the true prior manifold. To address this, we propose State-Guided Projected Gradient (SGPG): at each denoising step, the measurement gradient is *exactly* projected onto the low-rank subspace spanned by the current diffusion latent state, thereby preserving prior manifold fidelity. SGPG is the first method to dynamically guide gradient optimization using evolving diffusion states, significantly improving robustness to step size and noise level variations while enhancing worst-case performance. Compatible with diverse diffusion-based inversion frameworks, SGPG achieves state-of-the-art results across both linear and nonlinear image restoration tasks, effectively suppressing artifacts. Code is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning with Manifolds

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Responsible Web: Measurement, analysis, and circumvention of Web censorship

Abstract

Recent advancements in diffusion models have been effective in learning data priors for solving inverse problems. They leverage diffusion sampling steps for inducing a data prior while using a measurement guidance gradient at each step to impose data consistency. For general inverse problems, approximations are needed when an unconditionally trained diffusion model is used since the measurement likelihood is intractable, leading to inaccurate posterior sampling. In other words, due to their approximations, these methods fail to preserve the generation process on the data manifold defined by the diffusion prior, leading to artifacts in applications such as image restoration. To enhance the performance and robustness of diffusion models in solving inverse problems, we propose Diffusion State-Guided Projected Gradient (DiffStateGrad), which projects the measurement gradient onto a subspace that is a low-rank approximation of an intermediate state of the diffusion process. DiffStateGrad, as a module, can be added to a wide range of diffusion-based inverse solvers to improve the preservation of the diffusion process on the prior manifold and filter out artifact-inducing components. We highlight that DiffStateGrad improves the robustness of diffusion models in terms of the choice of measurement guidance step size and noise while improving the worst-case performance. Finally, we demonstrate that DiffStateGrad improves upon the state-of-the-art on linear and nonlinear image restoration inverse problems. Our code is available at https://github.com/Anima-Lab/DiffStateGrad.

Problem

• Improves diffusion models for inverse problems
• Reduces artifacts in image restoration
• Enhances robustness and performance of diffusion models

Innovation

• Projects gradient onto diffusion process subspace
• Enhances diffusion model robustness and performance
• Improves image restoration with artifact reduction