CoCo-IR: Contextual Composed Image Retrieval

2026-08-05

AI Summary

This work addresses the limitation of existing image retrieval systems, which are largely confined to single-turn queries and struggle to support complex, multi-round interactive search scenarios common in real-world applications. The study introduces Contextual Compositional Image Retrieval (CoCo-IR), a novel task formulation, and proposes a context-aware reasoning framework built upon large vision-language models. By modeling the full dialogue history, the framework generates evolvable Transformed Image Embeddings (TIE) to enable iterative refinement across multiple turns. Key contributions include the formal definition of the CoCo-IR task, the TIE mechanism, and a self-supervised data engine that automatically generates and validates training samples without human annotation. Experiments demonstrate that the method achieves 39.4 mAP@5 on the CIRCO single-turn benchmark and 44.1 R@1 on a newly curated four-turn CoCo-IR benchmark, substantially outperforming prior approaches, which attain only 28.2 R@1.

Abstract

Current instruction-based image retrieval systems are powerful but limited to single-turn interactions, failing to capture the iterative nature of complex, real-world visual searches. To overcome this limitation, we introduce Contextual Composed Image Retrieval (CoCo-IR), a novel task that enables users to progressively refine search results through interactions. We address this new task by proposing a new model based on a Large Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR. Our model interprets the entire interaction history to generate Transformable Image Embeddings (TIE) that evolve across turns. To fuel the model training without expensive human annotations, we develop a fully autonomous, scalable data engine that leverages LMMs to generate high-quality contextual retrieval data, and uses model-guided verification to mine challenging hard negatives. Extensive experiments demonstrate that our approach establishes new state-of-the-art performance: We achieve 39.4 mAP@5 on the challenging single-turn benchmark CIRCO; furthermore, on our new CoCo-IR benchmark, our model maintains robust performance with 44.1 R@1 on 4-turn dialogues, dramatically outperforming existing methods (28.2 4-turn R@1) that fail to handle multi-turn context. Project page: https://CoCo-IR.github.io.

Problem

• image retrieval
• multi-turn interaction
• contextual reasoning
• visual search
• interactive retrieval

Innovation

• Contextual Composed Image Retrieval
• Transformable Image Embeddings
• Large Multimodal Model
• Multi-turn Interaction
• Hard Negative Mining