Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusion

2025-11-16

AI Summary

Linear interpolation of prompt embeddings in Stable Diffusion—typically performed in Euclidean space—often yields abrupt, semantically discontinuous image transitions due to its oversimplified geometric assumption. Method: We propose a geometry-aware interpolation paradigm: interpreting CLIP text embedding matrices as point-cloud distributions in the Wasserstein space and computing their geodesic path via optimal transport theory, rather than applying conventional matrix-level linear interpolation. Contribution/Results: This is the first work to model prompt embeddings as probability distributions and leverage optimal transport for constructing semantically continuous interpolation flows. The method integrates seamlessly into the CLIP+Stable Diffusion pipeline without requiring model fine-tuning. Experiments demonstrate that Wasserstein interpolation significantly improves transition smoothness and semantic coherence in generated imagery, revealing the critical role of non-Euclidean geometry in embedding spaces for diffusion model performance. Our approach provides a novel perspective for analyzing and manipulating semantic representations in diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

It can be shown that Stable Diffusion has a permutation-invariance property with respect to the rows of Contrastive Language-Image Pretraining (CLIP) embedding matrices. This inspired the novel observation that these embeddings can naturally be interpreted as point clouds in a Wasserstein space rather than as matrices in a Euclidean space. This perspective opens up new possibilities for understanding the geometry of embedding space. For example, when interpolating between embeddings of two distinct prompts, we propose reframing the interpolation problem as an optimal transport problem. By solving this optimal transport problem, we compute a shortest path (or geodesic) between embeddings that captures a more natural and geometrically smooth transition through the embedding space. This results in smoother and more coherent intermediate (interpolated) images when rendered by the Stable Diffusion generative model. We conduct experiments to investigate this effect, comparing the quality of interpolated images produced using optimal transport to those generated by other standard interpolation methods. The novel optimal transport--based approach presented indeed gives smoother image interpolations, suggesting that viewing the embeddings as point clouds (rather than as matrices) better reflects and leverages the geometry of the embedding space.

Problem

• Analyzing embedding geometry's role in image interpolation for Stable Diffusion
• Reformulating interpolation as optimal transport between CLIP embeddings
• Evaluating geodesic paths for smoother transitions in generated images

Innovation

• Interpreting embeddings as point clouds in Wasserstein space
• Reframing interpolation as an optimal transport problem
• Computing geodesics for geometrically smooth image transitions