Diffusion Models are Secretly Exchangeable: Parallelizing DDPMs via Autospeculation

2025-05-06

AI Summary

While denoising diffusion probabilistic models (DDPMs) achieve high-quality generation, their sequential sampling process incurs prohibitive inference latency. Method: This work establishes, for the first time, the commutativity of DDPM noise increments under reparameterization—enabling Autospeculative Decoding (ASD), a novel parallel sampling framework that operates without auxiliary draft models and requires no architectural modifications or additional training. ASD leverages only the intrinsic structure of the pretrained DDPM, supporting near black-box deployment. Contribution/Results: We provide theoretical analysis proving an asymptotic parallel speedup of Õ(K^{1/3}), where K is the number of denoising steps. Empirical evaluation across image and audio generation tasks demonstrates substantial inference acceleration—up to 2.8× wall-clock speedup—while preserving generation fidelity with zero quality degradation. Our core contribution is the discovery of inherent commutativity in DDPMs and the consequent formulation of the first general-purpose, draft-model-free parallel sampling paradigm for diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Denoising Diffusion Probabilistic Models (DDPMs) have emerged as powerful tools for generative modeling. However, their sequential computation requirements lead to significant inference-time bottlenecks. In this work, we utilize the connection between DDPMs and Stochastic Localization to prove that, under an appropriate reparametrization, the increments of DDPM satisfy an exchangeability property. This general insight enables near-black-box adaptation of various performance optimization techniques from autoregressive models to the diffusion setting. To demonstrate this, we introduce emph{Autospeculative Decoding} (ASD), an extension of the widely used speculative decoding algorithm to DDPMs that does not require any auxiliary draft models. Our theoretical analysis shows that ASD achieves a $ ilde{O} (K^{frac{1}{3}})$ parallel runtime speedup over the $K$ step sequential DDPM. We also demonstrate that a practical implementation of autospeculative decoding accelerates DDPM inference significantly in various domains.

Problem

• DDPMs have sequential computation causing inference bottlenecks
• Prove DDPM increments satisfy exchangeability under reparametrization
• Introduce Autospeculative Decoding to accelerate DDPM inference

Innovation

• Exchangeable increments enable parallel DDPM optimization
• Autospeculative Decoding removes auxiliary draft models
• ASD achieves significant parallel runtime speedup