Cross-modal RAG: Sub-dimensional Retrieval-Augmented Text-to-Image Generation

2025-05-28

AI Summary

To address inaccurate text-to-image generation caused by knowledge inaccessibility under complex, fine-grained textual queries, this paper proposes a cross-modal sub-dimension decomposition framework that enables semantic sub-dimension–level disentangled alignment between queries and images. We introduce a novel sub-dimension retrieval augmentation paradigm, featuring a sub-query–aware hybrid sparse–dense retrieval strategy and a Pareto-optimal image set selection mechanism to support on-demand injection of multi-source visual features. Our method integrates sub-dimension–specific sparse retrieval, contrastive learning–driven cross-modal dense retrieval, and multimodal large language model–guided sub-query alignment generation. Evaluated on five benchmarks including MS-COCO, our approach significantly outperforms existing RAG-based methods: retrieval accuracy improves by 19.3%, FID decreases by 27.6%, and inference efficiency maintains linear scalability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Text-to-image generation increasingly demands access to domain-specific, fine-grained, and rapidly evolving knowledge that pretrained models cannot fully capture. Existing Retrieval-Augmented Generation (RAG) methods attempt to address this by retrieving globally relevant images, but they fail when no single image contains all desired elements from a complex user query. We propose Cross-modal RAG, a novel framework that decomposes both queries and images into sub-dimensional components, enabling subquery-aware retrieval and generation. Our method introduces a hybrid retrieval strategy - combining a sub-dimensional sparse retriever with a dense retriever - to identify a Pareto-optimal set of images, each contributing complementary aspects of the query. During generation, a multimodal large language model is guided to selectively condition on relevant visual features aligned to specific subqueries, ensuring subquery-aware image synthesis. Extensive experiments on MS-COCO, Flickr30K, WikiArt, CUB, and ImageNet-LT demonstrate that Cross-modal RAG significantly outperforms existing baselines in both retrieval and generation quality, while maintaining high efficiency.

Problem

• Text-to-image generation lacks fine-grained domain knowledge
• Existing RAG methods fail with complex multi-element queries
• Proposes sub-dimensional retrieval for query-aware image synthesis

Innovation

• Decomposes queries and images into sub-dimensional components
• Hybrid retrieval strategy combining sparse and dense retrievers
• Multimodal LLM guided for subquery-aware image synthesis