Specificity-Aware Diffusion Steering via Variance-Reduced Sequential Monte Carlo

2026-09-30

AI Summary

This study addresses the issue in diffusion model inference guidance where suppressing negative samples tends to distort the positive distribution. To overcome this, the authors formulate specificity guidance as a target design problem, deriving an overlap-based target distribution that retains reference signals exclusively within regions dominated by the positive distribution. Furthermore, they introduce a specificity objective function interpreted via likelihood ratios and develop a variance-minimizing locally-proposed Sequential Monte Carlo (SMC) sampler alongside a fixed-noise optimization procedure. The proposed approach effectively suppresses undesirable regions in tasks such as text-to-image generation, significantly mitigating mode shift and weight collapse while enhancing sampling stability.

Abstract

Inference-time steering enables pretrained diffusion models to satisfy new constraints without full retraining. However, specificity-aware generation is difficult: repelling samples from a negative reference distribution can also erode the positive distribution where the two overlap. The key challenge is to suppress negative mass while minimally distorting the positive distribution. We address this problem by formulating specificity-aware steering as a target-design problem and deriving a target distribution from an overlap-based objective. The resulting target keeps the desired reference distribution only in regions where it is sufficiently preferred over the undesired reference distribution, giving a likelihood-ratio interpretation of specificity. To sample from the corresponding time-dependent target path, we develop a Sequential Monte Carlo sampler with a variance-minimized local proposal. We further introduce a practical fixed-noise optimization procedure with the Jacobian--vector products with the desired and undesired score fields. Experiments on synthetic task, class-contrastive generation, text-to-image tasks and peptide-MHC (p-MHC) binder show that the proposed method suppresses undesired regions more effectively, reduces mode shift, and improves sampling stability by decreasing the SMC weight collapse compared with negative-guidance baselines. Code is available at: https://github.com/WangLuran/Specificity-Aware-Diffusion-Steering

Problem

• diffusion models
• inference-time steering
• specificity-aware generation
• distribution overlap
• negative guidance

Innovation

• Diffusion Steering
• Sequential Monte Carlo
• Variance Reduction
• Specificity-Aware Generation
• Likelihood Ratio