🤖 AI Summary
Existing RAG systems are limited to unimodal text retrieval and struggle to effectively process unstructured multimodal documents containing text, images, tables, mathematical formulas, and charts. To address this, we propose MAHA—a modality-aware hybrid retrieval architecture that innovatively integrates dense vector retrieval with knowledge graph–driven structured traversal. First, MAHA constructs a modality-aware knowledge graph that explicitly encodes cross-modal semantic relationships. Then, it jointly performs question answering and interpretable reasoning via synergistic multimodal embedding and graph traversal. Evaluated on multiple benchmark datasets, MAHA achieves state-of-the-art performance (e.g., ROUGE-L = 0.486), significantly outperforming baseline methods. It is the first approach to achieve full-modality coverage—supporting text, images, tables, formulas, and diagrams—while delivering both high retrieval coverage and strong interpretability. Comprehensive experiments validate MAHA’s effectiveness, robustness, and scalability across diverse multimodal document understanding tasks.
📝 Abstract
Current Retrieval-Augmented Generation (RAG) systems primarily operate on unimodal textual data, limiting their effectiveness on unstructured multimodal documents. Such documents often combine text, images, tables, equations, and graphs, each contributing unique information. In this work, we present a Modality-Aware Hybrid retrieval Architecture (MAHA), designed specifically for multimodal question answering with reasoning through a modality-aware knowledge graph. MAHA integrates dense vector retrieval with structured graph traversal, where the knowledge graph encodes cross-modal semantics and relationships. This design enables both semantically rich and context-aware retrieval across diverse modalities. Evaluations on multiple benchmark datasets demonstrate that MAHA substantially outperforms baseline methods, achieving a ROUGE-L score of 0.486, providing complete modality coverage. These results highlight MAHA's ability to combine embeddings with explicit document structure, enabling effective multimodal retrieval. Our work establishes a scalable and interpretable retrieval framework that advances RAG systems by enabling modality-aware reasoning over unstructured multimodal data.