SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis

2024-03-14

European Conference on Computer Vision

AI Summary

Existing diffusion models for semantic image synthesis (SIS) suffer from two key limitations: structural artifacts and misalignment between semantic masks and generated content. To address these, this paper proposes Spatial-Category Prior (SCP) modeling—a novel framework that explicitly couples spatial layout and category distribution for the first time, thereby mitigating training-inference noise distribution mismatch. Built upon the ControlNet architecture within the Latent Diffusion framework, our method jointly integrates spatial priors, category priors, and the SCP to enable fine-grained, semantics-controllable generation. Extensive experiments demonstrate state-of-the-art performance across three benchmark datasets: Cityscapes, ADE20K, and COCO-Stuff. Notably, on Cityscapes, our approach achieves an FID score of 10.53—marking a substantial improvement in structural consistency and semantic alignment accuracy.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Semantic image synthesis (SIS) shows good promises for sensor simulation. However, current best practices in this field, based on GANs, have not yet reached the desired level of quality. As latent diffusion models make significant strides in image generation, we are prompted to evaluate ControlNet, a notable method for its dense control capabilities. Our investigation uncovered two primary issues with its results: the presence of weird sub-structures within large semantic areas and the misalignment of content with the semantic mask. Through empirical study, we pinpointed the cause of these problems as a mismatch between the noised training data distribution and the standard normal prior applied at the inference stage. To address this challenge, we developed specific noise priors for SIS, encompassing spatial, categorical, and a novel spatial-categorical joint prior for inference. This approach, which we have named SCP-Diff, has set new state-of-the-art results in SIS on Cityscapes, ADE20K and COCO-Stuff, yielding a FID as low as 10.53 on Cityscapes. The code and models can be accessed via the project page.

Problem

• Addresses weird sub-structures in large semantic areas
• Resolves content misalignment with semantic masks
• Corrects mismatch between training noise and inference prior

Innovation

• Spatial-categorical joint prior for diffusion
• Specific noise priors for semantic synthesis
• Novel inference approach addressing distribution mismatch