Innovation
Methods, ideas, or system contributions that make the work stand out.
Retrieval-Augmented Generation
RAG-Stress
Misleading Rate
Evidence Reliance
Diagnostic Protocol
RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation
🤖 AI Summary
This study addresses the vulnerability of retrieval-augmented generation (RAG) systems to misleading evidence, which can cause models to override correct answers—a risk obscured by standard accuracy metrics. To investigate this, we propose RAG-Stress, a diagnostic protocol that decouples evidence compliance from factual reliability by fixing questions, perturbing retrieved evidence, and manipulating positional and prioritization strategies to quantify model susceptibility to misdirection. Evaluating fifteen systems across multiple question-answering datasets reveals that document-first prioritization strategies significantly increase misdirection rates by 10.9 to 13.5 percentage points. These findings demonstrate that instructing models to prioritize retrieved documents over parametric knowledge exacerbates harmful overrides without facilitating factual correction, highlighting a critical trade-off in current RAG implementations.
📝 Abstract
Following retrieved evidence does not guarantee factual correctness: misleading evidence can induce a model to replace an answer it previously gave correctly. Standard accuracy measures obscure this behavior by combining answer replacement with preexisting errors. We introduce RAG-Stress, a controlled diagnostic protocol for examining the limits of evidence reliance in retrieval-augmented generation. The protocol holds the question and reference answer fixed, edits one assertion to support a designated incorrect answer, and crosses two source priority policies with three positions of the answer span within the evidence text. We measure misleading rate (MR) on each model's subset of questions answered correctly without retrieval, alongside clean accuracy on the full evaluation set. We evaluate fifteen systems spanning API models, open models, and search agents trained with reinforcement learning on TriviaQA-RC, HotpotQA, and SearchQA, with additional English and Chinese MedQA evaluations. Instructions that prioritize documents consistently produce higher MR than those permitting reliance on prior knowledge. Averaged over models and positions, the gap ranges from 10.9 to 13.5 percentage points across the three QA datasets. Mean MR follows End $>$ Beginning $>$ Middle under both policies, although individual models do not uniformly follow this ordering. A separate paired audit of 500 questions and two checkpoints supports increased harmful override without establishing a corresponding improvement in beneficial correction. These findings distinguish evidence adherence from factual reliability and motivate evaluating whether retrieved evidence preserves, replaces, or corrects a model's answers.
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
