The Unreasonable Effectiveness of Guidance for Diffusion Models

2024-11-15

arXiv.org

AI Summary

This work addresses the challenge of classifier-free guidance in diffusion models—without additional training, access to historical weights, or class conditioning. We propose Sliding Window Guidance (SWG), which leverages the primary model itself as an auxiliary model, constraining its receptive field via a sliding window to enable spatial dependency modeling and alignment of error patterns. Our key insight is that the auxiliary model need not be highly accurate; it suffices for it to share error characteristics with the primary model while exhibiting stronger response sensitivity—yielding substantial gains in generation quality. We further introduce an error-driven guidance mechanism and weight regularization for stable control. SWG incurs zero training overhead, requires no architectural modifications or conditional inputs, and matches state-of-the-art guided methods on quantitative metrics (e.g., FID, LPIPS). Moreover, it achieves superior performance in human visual preference evaluations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Guidance is an error-correcting technique used to improve the perceptual quality of images generated by diffusion models. Typically, the correction is achieved by linear extrapolation, using an auxiliary diffusion model that has lower performance than the primary model. Using a 2D toy example, we show that it is highly beneficial when the auxiliary model exhibits similar errors as the primary one but stronger. We verify this finding in higher dimensions, where we show that competitive generative performance to state-of-the-art guidance methods can be achieved when the auxiliary model differs from the primary one only by having stronger weight regularization. As an independent contribution, we investigate whether upweighting long-range spatial dependencies improves visual fidelity. The result is a novel guidance method, which we call sliding window guidance (SWG), that guides the primary model with itself by constraining its receptive field. Intriguingly, SWG aligns better with human preferences than state-of-the-art guidance methods while requiring neither training, architectural modifications, nor class conditioning. The code will be released.

Problem

• Enhancing diffusion model sample quality without training
• Addressing oversaturation in training-free guidance methods
• Improving long-range spatial dependencies in image generation

Innovation

• Masked sliding window guidance method
• Training-free self-guidance technique
• Selective receptive field restriction