Is Relevance Propagated from Retriever to Generator in RAG?

2025-02-20

AI Summary

This study investigates whether topic relevance of retrieved documents in Retrieval-Augmented Generation (RAG) effectively improves downstream information retrieval performance. Moving beyond conventional “answer-containing” relevance, we propose and quantify a novel “topic-overlap” relevance dimension. Our methodology integrates the KILT benchmark, multi-scale context ablation experiments, standard retrievers (BM25, DPR), and utility attribution analysis. Results show that topic relevance exhibits only weak positive correlation with generation utility, and this correlation decays significantly as the number of retrieved documents (k) increases. While stronger retrieval models consistently improve overall RAG performance, they fail to translate topic relevance into utility gains linearly. The core contribution is the empirical and theoretical establishment of topic overlap as an independent relevance dimension, coupled with the discovery of a nonlinear attenuation law governing the relevance-to-utility transfer—highlighting diminishing returns in leveraging topic-relevant context as k grows.

Technology Category

Data Mining & Knowledge Management: Conversational Systems for Recommendation & RetrievalMachine Learning: Learning Preferences or RankingsKnowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingWeb Mining and Content Analysis: Topic discovery and tracking

Abstract

Retrieval Augmented Generation (RAG) is a framework for incorporating external knowledge, usually in the form of a set of documents retrieved from a collection, as a part of a prompt to a large language model (LLM) to potentially improve the performance of a downstream task, such as question answering. Different from a standard retrieval task's objective of maximising the relevance of a set of top-ranked documents, a RAG system's objective is rather to maximise their total utility, where the utility of a document indicates whether including it as a part of the additional contextual information in an LLM prompt improves a downstream task. Existing studies investigate the role of the relevance of a RAG context for knowledge-intensive language tasks (KILT), where relevance essentially takes the form of answer containment. In contrast, in our work, relevance corresponds to that of topical overlap between a query and a document for an information seeking task. Specifically, we make use of an IR test collection to empirically investigate whether a RAG context comprised of topically relevant documents leads to improved downstream performance. Our experiments lead to the following findings: (a) there is a small positive correlation between relevance and utility; (b) this correlation decreases with increasing context sizes (higher values of k in k-shot); and (c) a more effective retrieval model generally leads to better downstream RAG performance.

Problem

• Assessing relevance propagation in RAG systems
• Exploring topical overlap impact on RAG utility
• Investigating retrieval model effectiveness in RAG performance

Innovation

• RAG integrates external knowledge
• Utility maximized over relevance
• Topical overlap improves performance