A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

2026-08-05

AI Summary

This study addresses the limitations of existing vision–language retrieval models, which predominantly rely on short captions and struggle with long-text retrieval tasks. The authors systematically investigate the impact of text granularity—ranging from single-sentence captions to multi-sentence paragraphs—on contrastive image–text retrieval performance and propose an efficient training strategy that fine-tunes only the text encoder while freezing the visual encoder. Leveraging a synthetic data pipeline built upon Qwen2-VL and Llama 3.2 Vision, they generate diverse captions, hard negatives, and scored paragraphs, revealing for the first time that paragraph-level supervision substantially enhances long-text retrieval. Experiments demonstrate that the proposed method improves image-to-text retrieval R@1 by over 14 points on DOCCI, outperforming Long-CLIP-L and matching ShareGPT4V performance, while effectively mitigating the performance degradation observed with caption-based training beyond 60 tokens.

Abstract

Contrastive vision-language models such as CLIP and BLIP are typically trained on short image captions, limiting their ability to retrieve images from detailed textual descriptions. While methods such as Long-CLIP extend the token limit through positional embedding interpolation, we ask a simpler question: does training text granularity alone determine long-text retrieval performance? We present a systematic study of supervision ranging from single captions to multi-sentence paragraphs for contrastive image-text retrieval. Using a synthetic pipeline based on Qwen2-VL and Llama 3.2 Vision, we generate diverse captions, hard negatives, and quality-scored paragraphs for 500K CC3M images. To isolate the effect of text granularity, we fine-tune only the BLIP text encoder while keeping the vision encoder frozen across 10 training configurations. Our paragraph-supervised models match Long-CLIP-L on ShareGPT4V and outperform it by more than 14 points on DOCCI for image-to-text retrieval, without architectural changes. We further show that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens. Increasing caption diversity improves short-caption retrieval with diminishing returns, while paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning. Evaluations on Flickr30k, COCO, ShareGPT4V, and DOCCI provide a comprehensive analysis of the trade-offs between text granularity, retrieval direction, and description length.

Problem

• vision-language retrieval
• text supervision
• long-text retrieval
• caption granularity
• contrastive learning

Innovation

• paragraph-level supervision
• vision-language retrieval
• text granularity
• contrastive learning
• long-text understanding