Controlling Embedding Spaces with Text-Conditioned Transformations

2026-07-24

AI Summary

Existing multimodal embedding spaces, such as CLIP, tend to prioritize dominant objects during semantic compression while suppressing fine-grained attributes like viewpoint or color tone, thereby limiting precise retrieval and manipulation. To address this, this work proposes a text-conditioned affine transformation network that dynamically modulates visual features within a frozen embedding space, enabling explicit disentanglement and emphasis of any user-specified attribute—such as color or artistic style—without requiring re-encoding. This approach achieves, for the first time, real-time control over the embedding space via natural language instructions at zero inference cost, supporting structured organization and retrieval across multiple attributes. Experiments demonstrate state-of-the-art performance on attribute retrieval and multi-attribute clustering tasks, while maintaining high computational efficiency and compatibility with existing models.

Abstract

Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level semantics into a single vector, which comes at the cost of primarily expressing a dominant semantics like main object while suppressing other important attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., "color" or "art style"), a network generates an affine transformation that emphasizes the specified attribute. Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost. Project page: https://joefioresi718.github.io/ControlEmbed_webpage/

Problem

• multimodal embedding
• attribute suppression
• semantic compression
• embedding space control
• latent space disentanglement

Innovation

• text-conditioned transformation
• embedding space control
• attribute disentanglement
• zero-shot retrieval
• latent space manipulation