Visual Semantic Description Generation with MLLMs for Image-Text Matching

2025-07-11

AI Summary

Image–text matching faces a fundamental challenge stemming from the semantic gap between visual (continuous, high-dimensional) and textual (discrete, structured) modalities. To address this, we propose VSD-Align, a novel framework that leverages multimodal large language models (MLLMs) to generate fine-grained Visual Semantic Descriptions (VSDs) as learnable semantic anchors. VSD-Align establishes dual alignment: instance-level feature fusion and prototype-level VSD clustering—enabling cross-modal semantic alignment at the representation level. The method supports end-to-end training without auxiliary annotations and inherently enables zero-shot cross-domain transfer. Extensive experiments demonstrate that VSD-Align significantly outperforms state-of-the-art methods on Flickr30K and MSCOCO. Moreover, it achieves superior generalization in zero-shot cross-domain settings—including news and remote sensing domains—validating its effectiveness and broad applicability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Image-text matching (ITM) aims to address the fundamental challenge of aligning visual and textual modalities, which inherently differ in their representations, continuous, high-dimensional image features vs. discrete, structured text. We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantic parsers. By generating rich Visual Semantic Descriptions (VSD), MLLMs provide semantic anchor that facilitate cross-modal alignment. Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency. These modules can be seamlessly integrated into existing ITM models. Extensive experiments on Flickr30K and MSCOCO demonstrate substantial performance improvements. The approach also exhibits remarkable zero-shot generalization to cross-domain tasks, including news and remote sensing ITM. The code and model checkpoints are available at https://github.com/Image-Text-Matching/VSD.

Problem

• Bridging visual-textual modality gap using MLLMs
• Enhancing cross-modal alignment via semantic descriptions
• Improving image-text matching with zero-shot generalization

Innovation

• Uses MLLMs as visual semantic parsers
• Generates Visual Semantic Descriptions (VSD)
• Combines instance and prototype level alignment