When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG

2026-06-02

AI Summary

This study systematically evaluates the practical efficacy of Retrieval-Augmented Generation (RAG) in high-stakes biomedical question answering. Through large-scale, multidimensional experiments across multiple open-source large language models (7B to 72B parameters), four retrieval methods, four corpora, and ten biomedical QA datasets, we find that RAG yields only marginal performance gains—averaging 1–2 percentage points—substantially less than the impact of model architecture choice. Notably, specialized and non-specialized retrieval sources perform similarly. Crucially, this work reveals for the first time that RAG’s limitations in this domain stem primarily from the model’s inadequate ability to leverage retrieved evidence, rather than deficiencies in retrieval quality itself, thereby underscoring the critical need to enhance model reasoning mechanisms.

Abstract

Medical question answering is a high-stakes setting where factual errors can have serious consequences. Retrieval-augmented generation (RAG) is widely viewed as a promising solution, and prior work has reported substantial gains for large medical QA models. We revisit this assumption across a broad range of open-weight instruction-tuned models spanning 7B to 72B parameters. Across five models, ten biomedical QA datasets, four retrieval methods, and four retrieval corpora, we find that retrieval yields only small and inconsistent improvements over a no-retrieval baseline, typically within 1-2 points. In contrast, the choice of backbone model has a much larger effect than the choice of retriever or corpus, and expert and layman retrieval sources perform similarly in most settings. These results suggest that the main bottleneck is not retrieval quality alone, but the model's limited ability to use retrieved evidence effectively.

Problem

• biomedical question answering
• retrieval-augmented generation
• RAG
• factual errors
• retrieval effectiveness

Innovation

• retrieval-augmented generation
• biomedical question answering
• large language models
• retrieval effectiveness
• evidence utilization