Testing Retrieval-Augmented Generation Systems with Chunk Coverage

2026-07-20

AI Summary

This work addresses the limitations of existing RAG system evaluations, which rely on query-level annotations or reference answers and thus fail to comprehensively assess the behavioral coverage of retrieval components across test suites. To overcome this, we propose Chunk Coverage (CC), a novel metric that—without requiring test oracles—quantifies retrieval coverage by measuring the proportion of corpus chunks retrieved at least once during testing, thereby characterizing coverage at the structural level. Building upon CC, we develop an unsupervised framework for evaluating retrieval behavior and guiding test case selection and prioritization. Empirical results in clinical and financial domains demonstrate that CC-guided testing achieves 50% coverage 1.7× faster than random strategies and 4.2× faster than redundancy-based approaches, while improving fault detection capability (measured by APFD) by 10%–25%.

Abstract

Retrieval-Augmented Generation (RAG)-based systems\footnote{For brevity, RAG-based systems are referred to as RAG systems throughout this paper.} are increasingly deployed in high-stakes settings where correct behaviour depends not only on the language model but also on the retrieval component that selects external documents at inference time. While existing RAG evaluation metrics assess retrieval and generation quality on a per-query basis, typically relying on query-level test oracles such as reference answers or relevance annotations, they provide limited insight into whether a test suite adequately exercises the retrieval behaviour of the system as a whole. In this paper, we introduce Chunk Coverage (CC), an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems. CC measures the fraction of corpus chunks that are retrieved at least once across a test suite, providing a structural view of which parts of the retrieval space have been exercised. We further show how CC can be used to guide test selection and generation by prioritising queries that expand coverage of previously unexercised retrieval regions. We evaluate CC on clinical and financial RAG system scenarios. CC-guided testing reaches 50% of attainable coverage 1.7x faster than random selection and 4.2x faster than redundancy-biased strategies. Moreover, CC improves fault detection effectiveness (APFD) by 10% to 25% over random, indicating earlier discovery of distinct retrieval faults. These results show that CC captures retrieval diversity relevant to effective testing without requiring test oracles.

Problem

• Retrieval-Augmented Generation
• test adequacy
• retrieval coverage
• oracle-independent evaluation
• chunk coverage

Innovation

• Chunk Coverage
• Retrieval-Augmented Generation
• Test Adequacy
• Oracle-Independent Testing
• Retrieval Diversity