🤖 AI Summary
Existing multimodal RAG systems rely on LLMs to generate textual summaries of images and index only these summaries, leading to loss of critical visual details and contextual information—particularly detrimental to chart-text joint question answering in financial documents. This work proposes a direct multimodal embedding retrieval approach: leveraging models such as CLIP to jointly encode images and text into a shared embedding space, enabling cross-modal vector retrieval without intermediate LLM summarization and its associated information decay. Evaluated on a newly constructed financial report QA benchmark, our method achieves a 13-percentage-point absolute gain in mAP@5 (32% relative improvement) and an 11-percentage-point gain in nDCG@5 (20% relative improvement) over the text-summary baseline, significantly enhancing retrieval relevance and factual consistency of generated answers. Extensive experiments across six mainstream LLMs demonstrate the robustness and generalizability of the proposed approach.
📝 Abstract
Recent advancements in Retrieval-Augmented Generation (RAG) have enabled Large Language Models (LLMs) to access multimodal knowledge bases containing both text and visual information such as charts, diagrams, and tables in financial documents. However, existing multimodal RAG systems rely on LLM-based summarization to convert images into text during preprocessing, storing only text representations in vector databases, which causes loss of contextual information and visual details critical for downstream retrieval and question answering. To address this limitation, we present a comprehensive comparative analysis of two retrieval approaches for multimodal RAG systems, including text-based chunk retrieval (where images are summarized into text before embedding) and direct multimodal embedding retrieval (where images are stored natively in the vector space). We evaluate all three approaches across 6 LLM models and a two multi-modal embedding models on a newly created financial earnings call benchmark comprising 40 question-answer pairs, each paired with 2 documents (1 image and 1 text chunk). Experimental results demonstrate that direct multimodal embedding retrieval significantly outperforms LLM-summary-based approaches, achieving absolute improvements of 13% in mean average precision (mAP@5) and 11% in normalized discounted cumulative gain. These gains correspond to relative improvements of 32% in mAP@5 and 20% in nDCG@5, providing stronger evidence of their practical impact. We additionally find that direct multimodal retrieval produces more accurate and factually consistent answers as measured by LLM-as-a-judge pairwise comparisons. We demonstrate that LLM summarization introduces information loss during preprocessing, whereas direct multimodal embeddings preserve visual context for retrieval and inference.