FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval

2026-07-30

AI Summary

This work addresses the limitations of existing approaches in complex image retrieval, which suffer from insufficient fine-grained contextual modeling and entangled optimization objectives, thereby constraining the performance of multimodal large language models. To overcome these challenges, the authors propose an automatic pipeline for constructing a fine-grained multimodal quintuple dataset and introduce a two-stage decoupled fine-tuning strategy: first enhancing contextual reasoning capabilities and subsequently refining retrieval alignment. The proposed method achieves substantial improvements over current state-of-the-art approaches across five complex image retrieval benchmarks. Notably, even with a lightweight backbone model under zero-shot settings, it attains leading performance, demonstrating the efficacy of fine-grained representation learning and staged optimization in multimodal retrieval tasks.

Abstract

Due to their strong generalizable multimodal processing and reasoning capabilities, Multimodal Large Language Models (MLLMs) have demonstrated significant potential as universal image retrievers, effectively addressing diverse real-world image retrieval tasks. Nevertheless, pioneering studies, while promising, overlook the potential of fine-grained context modeling and disentangled fine-tuning objectives in enhancing MLLMs' retrieval performance, particularly for complex tasks such as long-text-to-image retrieval, visual dialog retrieval, and composed image retrieval (CIR). Therefore, in this work, we propose an automated fine-grained multimodal quintuple dataset construction pipeline and a novel two-stage fine-grained multimodal fine-tuning strategy. The dataset generation pipeline produces a comprehensive CIR dataset with fine-grained image captions and modification text, facilitating fine-grained context modeling. Beyond the previously entangled fine-tuning paradigm, our approach separates the fine-tuning process into two distinct stages: (1) fine-grained context reasoning-oriented fine-tuning and (2) fine-grained retrieval-oriented fine-tuning. These stages aim to sequentially enhance the model's context understanding and query-target alignment capabilities, thereby improving retrieval performance. Extensive experiments across five datasets encompassing diverse and complex image retrieval tasks demonstrate the remarkable superiority of our method over existing approaches in zero-shot retrieval settings, even with a more lightweight MLLM backbone compared to those methods.

Problem

• Multimodal Large Language Models
• Complex Image Retrieval
• Fine-Grained Context Modeling
• Composed Image Retrieval
• Visual Dialog Retrieval

Innovation

• fine-grained context learning
• multimodal large language models
• composed image retrieval
• two-stage fine-tuning
• context reasoning