Does More Retrieved Evidence Help Visual Retrieval-Augmented Generation with Diffusion Language Models?

2026-08-07

AI Summary

In visual retrieval-augmented generation, indiscriminately incorporating more retrieved evidence, while improving recall, often degrades the answer accuracy of diffusion language models due to semantic conflicts. This work identifies that mixing incompatible visual sources during parallel denoising leads to answer distortion and proposes a training-free Entropy-based Candidate Filter (ECF) to selectively introduce high-quality evidence. ECF leverages multi-granularity evidence units and confidence estimation from blank control blocks to assess retrieval candidates. Integrated with retrieval re-ranking and a training-free admission mechanism, the method achieves an average accuracy gain of 2.62 percentage points across three visual diffusion language models and five visual question answering benchmarks, significantly outperforming fixed top-k selection and other training-free strategies.

Abstract

Visual retrieval-augmented generation (RAG) commonly expands the retrieved evidence set to improve answer-page coverage, implicitly assuming that all available evidence should be passed to the generator. We show that this assumption does not hold for diffusion language models (DLMs): retrieving more pages increases answer-page recall, whereas unconditionally passing all retrieved pages to the generator often reduces answer accuracy, primarily because of semantic conflict. A latent-source analysis explains this mismatch through source-coherence loss in parallel denoising, where position-wise proposals can combine incompatible visual sources into unsupported answers. We further find that such interference is already visible in the first-step answer-block distribution, making it possible to assess evidence before decoding. To preserve retrieval coverage while limiting harmful visual exposure, we propose the Entropy-Based Candidate Filter (ECF), a training-free evidence-admission framework. To reduce irrelevant content within individual candidates, ECF constructs multi-granularity evidence units; to identify beneficial additional evidence, it uses blank-controlled block confidence and retrieval rank to determine whether and which candidate should enter the final context. Across three multimodal DLMs and five visual QA benchmarks, ECF improves answer accuracy by 2.62 percentage points on average over the strongest fixed top-$k$ input and, with LLaDA2.0-Uni, by 2.37 percentage points on average over the best competing training-free result for each dataset. These results show that broader retrieval benefits visual DLM-RAG through selective evidence admission rather than unconditional evidence expansion. Code is publicly available at https://github.com/wjkuser/ECF.

Problem

• visual retrieval-augmented generation
• diffusion language models
• semantic conflict
• evidence selection
• answer accuracy

Innovation

• Diffusion Language Models
• Visual Retrieval-Augmented Generation
• Entropy-Based Candidate Filter
• Source-Coherence Loss
• Evidence Selection