When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations

2026-07-29

AI Summary

This work addresses a critical limitation in current RAG system evaluation, which relies on static snapshots and fails to capture errors induced by knowledge base evolution. The authors propose the first metamorphic testing framework tailored for RAG systems, formally defining failure types under knowledge changes and introducing 11 mutation operators operating at both indexing and retrieval context levels. They develop a high-precision metamorphic oracle that integrates pre- and post-chunking mutations, LLM-based reranking, and generator upgrades. Evaluated across five datasets with over 28,000 mutated samples, the approach reveals metamorphic violation rates of 4.9–10.2% and achieves oracle F1 scores ranging from 0.927 to 1.000—substantially outperforming RAGAS, whose F1 score peaks at 0.570.

Abstract

Retrieval-Augmented Generation (RAG)-based LLM systems rely on external document corpora that can evolve and change over time. However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data. We introduce a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution. We formalise a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels. An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%. In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570. Finally, we provide actionable insights into mitigating these faults through retrieval re-configuration, generator upgrades, and LLM-based reranking.

Problem

• Retrieval-Augmented Generation
• metamorphic testing
• corpus evolution
• evaluation methodology
• fault detection

Innovation

• Metamorphic Testing
• RAG Systems
• Corpus Evolution
• Mutation Operators
• Consistency Evaluation