Understanding and Mitigating Compositional Issues in Text-to-Image Generative Models

2024-06-12

arXiv.org

AI Summary

This work addresses the attribute-object-relation compositional misalignment problem in text-to-image generation, revealing structural deficiencies in the CLIP text encoder: its final-layer token embeddings are susceptible to interference from irrelevant words, and its attention mechanism struggles to capture fine-grained semantic relations. We propose a lightweight solution—fine-tuning only the CLIP linear projection head—without modifying the encoder backbone or the diffusion model architecture. On benchmarks such as Compositional COCO, our method improves compositional accuracy by 23.6% while preserving FID. Through attention reweighting and representational space analysis, we empirically demonstrate that suboptimal geometry in the CLIP embedding space is the primary cause of compositional failure. To our knowledge, this is the first systematic study establishing that projection-head fine-tuning alone suffices to substantially enhance compositional generalization—introducing a novel, efficient paradigm for improving multimodal alignment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent text-to-image diffusion-based generative models have the stunning ability to generate highly detailed and photo-realistic images and achieve state-of-the-art low FID scores on challenging image generation benchmarks. However, one of the primary failure modes of these text-to-image generative models is in composing attributes, objects, and their associated relationships accurately into an image. In our paper, we investigate this compositionality-based failure mode and highlight that imperfect text conditioning with CLIP text-encoder is one of the primary reasons behind the inability of these models to generate high-fidelity compositional scenes. In particular, we show that (i) there exists an optimal text-embedding space that can generate highly coherent compositional scenes which shows that the output space of the CLIP text-encoder is sub-optimal, and (ii) we observe that the final token embeddings in CLIP are erroneous as they often include attention contributions from unrelated tokens in compositional prompts. Our main finding shows that the best compositional improvements can be achieved (without harming the model's FID scores) by fine-tuning {it only} a simple linear projection on CLIP's representation space in Stable-Diffusion variants using a small set of compositional image-text pairs. This result demonstrates that the sub-optimality of the CLIP's output space is a major error source. We also show that re-weighting the erroneous attention contributions in CLIP can also lead to improved compositional performances, however these improvements are often less significant than those achieved by solely learning a linear projection head, highlighting erroneous attentions to be only a minor error source.

Problem

• Addresses attribute binding failures in text-to-image models
• Identifies sub-optimal CLIP text-encoder embeddings as a key issue
• Proposes fine-tuning linear projection for improved compositional generation

Innovation

• Fine-tuning CLIP's linear projection for better embeddings
• Optimizing text-embedding space for compositional scenes
• Correcting final token embeddings in CLIP attention