Spectral Prior for Reducing Exposure Bias in Diffusion Models

2026-07-24

AI Summary

This work addresses the exposure bias and error accumulation in diffusion models during inference, which stem from a mismatch between the frequency-domain distributions of training and sampling phases. The study systematically identifies that this issue arises from structural discrepancies in spectral signal-to-noise ratios and introduces Spectral Alignment (SPA), a novel method that aligns these distributions without altering the training procedure. SPA leverages offline, data-driven spectral priors and employs FFT-based gradient guidance during inference to calibrate the power spectrum of intermediate predictions. The approach is lightweight, architecture-agnostic, and fully compatible with Classifier-Free Guidance. Evaluated across diverse models—including DDPM, ADM, Stable Diffusion 2.0, SDXL, SD3.5, and FLUX—SPA consistently enhances generation quality with only a 3–4% increase in computational overhead.

Abstract

Diffusion models typically suffer from error accumulation during iterative sampling, commonly referred to as exposure bias. We reveal systematic frequency-dependent discrepancies between training and inference, which can be interpreted as frequency-dependent SNR error. Crucially, the direction of this mismatch varies across models and timesteps, indicating that fixed correction rules do not generalize. We propose Spectral Alignment (SPA), a lightweight, guidance-based method that calibrates the power spectrum of intermediate predictions to a pre-computed prior. Our approach consists of two stages: (1) offline fitting of a parametric spectrum model from training data, and (2) inference-time guidance via efficient FFT-based gradient computation. SPA introduces minimal computational overhead (3-4\%) and is complementary to Classifier-Free Guidance (CFG). We demonstrate consistent improvements across diverse architectures, from pixel-space models (DDPM, ADM) to latent diffusion models (SD2.0, SDXL) and flow-matching models (SD3.5, FLUX). Our implementation is available at https://github.com/SonyResearch/SPA.

Problem

• exposure bias
• diffusion models
• frequency discrepancy
• SNR error
• iterative sampling

Innovation

• Spectral Alignment
• Exposure Bias
• Diffusion Models
• Frequency-dependent SNR
• FFT-based Guidance