Diffusion Model with Perceptual Loss

2023-12-30

arXiv.org

AI Summary

Diffusion models suffer from severe image distortion under classifier-free guidance, primarily due to the misalignment between conventional mean-squared-error (MSE) loss and human visual perception. To address this, we propose a perceptually consistent self-supervised loss function, offering the first perceptual-supervision perspective on the efficacy of guidance mechanisms. Specifically, during the diffusion process, we construct self-supervised targets using deep features extracted from a pre-trained VGG network—requiring neither auxiliary classifiers nor dedicated guidance networks. Our method is fully compatible with standard diffusion frameworks and significantly improves generation quality even in the zero-guidance regime: FID scores drop substantially, human evaluation scores rise markedly, and the long-standing quality-diversity trade-off is alleviated. Generated images exhibit enhanced photorealism and richer fine-grained detail.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Deep Generative Models & AutoencodersNatural Language Processing: Generation

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: LLM based quality controls for crowd workGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models without guidance generate very unrealistic samples. Guidance is used ubiquitously, and previous research has attributed its effect to low-temperature sampling that improves quality by trading off diversity. However, this perspective is incomplete. Our research shows that the choice of the loss objective is the underlying reason raw diffusion models fail to generate desirable samples. In this paper, (1) our analysis shows that the loss objective plays an important role in shaping the learned distribution and the MSE loss derived from theories holds assumptions that misalign with data in practice; (2) we explain the effectiveness of guidance methods from a new perspective of perceptual supervision; (3) we validate our hypothesis by training a diffusion model with a novel self-perceptual loss objective and obtaining much more realistic samples without the need for guidance. We hope our work paves the way for future explorations of the diffusion loss objective.

Problem

• Explores why raw diffusion models fail to generate realistic samples.
• Analyzes the role of loss objectives in shaping learned distributions.
• Proposes a novel self-perceptual loss for realistic sample generation.

Innovation

• Introduces perceptual loss in diffusion models
• Replaces MSE loss with self-perceptual loss
• Generates realistic samples without guidance