Technology Category
Computer Vision: Diffusion Models for VisionMachine Learning: Deep Generative Models & AutoencodersNatural Language Processing: Generation
ELROND: Exploring and decomposing intrinsic capabilities of diffusion models
🤖 AI Summary
Diffusion models struggle to achieve fine-grained semantic control under fixed text prompts due to inherent stochasticity. To address this, this work proposes directly disentangling semantic directions in the input embedding space. By analyzing gradient differences across multiple generations from the same prompt and integrating backpropagation, principal component analysis, and sparse autoencoders, the method constructs an interpretable and controllable semantic subspace. This approach enables, for the first time, precise intervention on individual concepts within the embedding space, effectively mitigating mode collapse in distilled models. Furthermore, it introduces a novel metric of concept complexity based on subspace dimensionality, offering a quantitative measure of a model’s capacity to represent specific semantics.
Technology Category
Application Category
📝 Abstract
A single text prompt passed to a diffusion model often yields a wide range of visual outputs determined solely by stochastic process, leaving users with no direct control over which specific semantic variations appear in the image. While existing unsupervised methods attempt to analyze these variations via output features, they omit the underlying generative process. In this work, we propose a framework to disentangle these semantic directions directly within the input embedding space. To that end, we collect a set of gradients obtained by backpropagating the differences between stochastic realizations of a fixed prompt that we later decompose into meaningful steering directions with either Principal Components Analysis or Sparse Autoencoder. Our approach yields three key contributions: (1) it isolates interpretable, steerable directions for precise, fine-grained control over a single concept; (2) it effectively mitigates mode collapse in distilled models by reintroducing lost diversity; and (3) it establishes a novel estimator for concept complexity under a specific model, based on the dimensionality of the discovered subspace.
Problem
Research questions and friction points this paper is trying to address.
diffusion models
semantic variation
stochastic generation
user control
generative process
Innovation
Methods, ideas, or system contributions that make the work stand out.
diffusion models
semantic disentanglement
steerable directions
mode collapse mitigation
concept complexity
🔎 Similar Papers
No similar papers found.
