Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study

2025-02-27

AI Summary

This work identifies a dual bias mechanism in CLIP for complex multi-object scenes: the image encoder favors larger objects, while the text encoder favors objects appearing earlier in the caption. To isolate these biases, we construct two controlled high-resolution datasets—SimCO and CompCO—enabling the first controlled experimental decoupling of encoder-specific biases. By integrating COCO training distribution analysis with CLIP training dynamics tracing, we establish their evolutionary correlation. We further validate their substantial impact on Stable Diffusion’s text-to-image generation. On a custom multi-object benchmark, encoder behavioral attribution and cross-model transfer analysis quantitatively demonstrate that size and word-order biases degrade image–text matching and generation performance by up to 42%. The study introduces the first reproducible diagnostic framework for assessing multi-object robustness in vision-language models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Contrastive Language-Image Pre-training (CLIP) models have demonstrated remarkable performance in zero-shot classification tasks, yet their efficacy in handling complex multi-object scenarios remains challenging. This study presents a comprehensive analysis of CLIP's performance limitations in multi-object contexts through controlled experiments. We introduce two custom datasets, SimCO and CompCO, to evaluate CLIP's image and text encoders in various multi-object configurations. Our findings reveal significant biases in both encoders: the image encoder favors larger objects, while the text encoder prioritizes objects mentioned first in descriptions. We hypothesize these biases originate from CLIP's training process and provide evidence through analyses of the COCO dataset and CLIP's training progression. Additionally, we extend our investigation to Stable Diffusion models, revealing that biases in the CLIP text encoder significantly impact text-to-image generation tasks. Our experiments demonstrate how these biases affect CLIP's performance in image-caption matching and generation tasks, particularly when manipulating object sizes and their order in captions. This work contributes valuable insights into CLIP's behavior in complex visual environments and highlights areas for improvement in future vision-language models.

Problem

• Analyzes CLIP's limitations in multi-object scenarios.
• Identifies biases in CLIP's image and text encoders.
• Examines CLIP's impact on text-to-image generation tasks.

Innovation

• Custom datasets SimCO, CompCO
• Analyzes CLIP encoder biases
• Extends to Stable Diffusion models