How can embedding models bind concepts?

2026-05-29

AI Summary

This work addresses the failure of existing vision-language models, such as CLIP, to correctly bind attributes like color and shape in multi-object scenes, which undermines compositional generalization. Through binding function analysis of embedding structures, we reveal that CLIP’s high-complexity binding mechanism impedes effective cross-modal alignment. To overcome this limitation, we introduce a low-complexity, multiplicative interaction-based binding function within a controlled Transformer architecture, trained on synthetic data and augmented with cross-modal probing techniques. Our experiments demonstrate that this approach spontaneously learns binding mechanisms capable of systematic generalization, significantly outperforming CLIP on unseen concept compositions. The implementation is publicly released.

Abstract

Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP's binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization. Code is publicly available at https://github.com/oshapio/binding-concepts-complexity.

Problem

• concept binding
• vision-language models
• systematic generalization
• embedding models
• object representation

Innovation

• concept binding
• embedding models
• systematic generalization
• multiplicative interactions
• binding function complexity