CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

2026-10-02

AI Summary

This study addresses the challenges of evidence redundancy and the absence of confidence control in multimodal retrieval-augmented generation by proposing a training-free inference framework. The proposed approach introduces a novel integration of complementary selection based on Maximal Marginal Relevance (MMR) with a confidence-guided mechanism leveraging Retriever/Evidence/Critic (R/E/C) scoring. This design constructs a complementary evidence pool and iteratively refines generated answers without modifying underlying model parameters. Experimental results demonstrate that the method yields significant performance improvements on benchmarks such as Encyclopedic-VQA, while ablation studies validate the effectiveness of each individual component.

Abstract

Multimodal large language models (MLLMs) have shown strong visual reasoning abilities, but knowledge-intensive visual question answering often requires external textual evidence beyond the image and the model's parametric knowledge. Existing multimodal RAG systems commonly rely on Top-$K$ retrieval or reranking, which may return redundant passages and provide limited control over whether an answer update is sufficiently supported by the retrieved evidence. We propose \textit{CLIMB}, a training-free inference-time framework for multimodal RAG. CLIMB first constructs a compact complementary evidence pool using an MMR-style objective that balances query relevance and passage-level redundancy. It then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when the estimated confidence increases. This design provides a simple stopping criterion and reduces unnecessary refinement without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show that CLIMB consistently improves over retrieval-augmented multimodal baselines. Ablations further indicate that complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute to the final performance.

Problem

• Multimodal Retrieval-Augmented Generation
• Visual Question Answering
• Redundant Passages
• Evidence Sufficiency
• Confidence Control

Innovation

• Multimodal RAG
• Training-free framework
• Complementary evidence pool
• Confidence-controlled refinement
• Cross-modal alignment