Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo

2025-02-09

AI Summary

Diffusion models face challenges in image inverse problems, including difficulty in conditional sampling and latent-space encoder-decoder mismatch. To address these, this paper proposes a latent-space conditional sampling framework based on Sequential Monte Carlo (SMC). It is the first work to systematically integrate SMC into the latent space of diffusion models: auxiliary observations are injected during the forward diffusion process, and a VAE-based encoder-decoder architecture is combined with observation-guided resampling to improve posterior sampling accuracy and diversity. The method requires no fine-tuning or additional training and is fully compatible with standard diffusion priors. Extensive experiments on ImageNet and FFHQ for super-resolution, denoising, and compressive sensing demonstrate consistent superiority over existing diffusion-based approaches—achieving higher PSNR and lower LPIPS scores—thereby validating its efficiency and strong generalization capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In image processing, solving inverse problems is the task of finding plausible reconstructions of an image that was corrupted by some (usually known) degradation model. Commonly, this process is done using a generative image model that can guide the reconstruction towards solutions that appear natural. The success of diffusion models over the last few years has made them a leading candidate for this task. However, the sequential nature of diffusion models makes this conditional sampling process challenging. Furthermore, since diffusion models are often defined in the latent space of an autoencoder, the encoder-decoder transformations introduce additional difficulties. Here, we suggest a novel sampling method based on sequential Monte Carlo (SMC) in the latent space of diffusion models. We use the forward process of the diffusion model to add additional auxiliary observations and then perform an SMC sampling as part of the backward process. Empirical evaluations on ImageNet and FFHQ show the benefits of our approach over competing methods on various inverse problem tasks.

Problem

• Inverse problem sampling in latent space
• Challenges in sequential diffusion models
• Novel SMC method for image reconstruction

Innovation

• Sequential Monte Carlo sampling
• Latent space utilization
• Diffusion model enhancement