When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse

2026-08-07

AI Summary

This work addresses the vulnerability of Retrieval-Augmented Generation (RAG) systems to poisoning attacks, wherein adversaries inject malicious documents to manipulate model outputs. Conventional detection methods based on output uncertainty often fail under such attacks due to artificially induced high confidence in incorrect responses. The study reveals, for the first time, that RAG poisoning attacks trigger an attention collapse phenomenon in the generator—characterized by a sharp drop in attention entropy and excessive concentration on poisoned documents. Building on this insight, the authors propose D-SCAN, a lightweight detection framework that monitors internal document-level attention dynamics without requiring access to the final output. Experiments demonstrate that D-SCAN substantially outperforms existing approaches across multiple attack benchmarks and remains effective even when the attack leaves the final answer unchanged.

Abstract

Retrieval-augmented generation (RAG) is indispensable for enhancing large language models. However, RAGs are increasingly susceptible to poisoning attacks, in which adversarial documents are injected to manipulate generator outputs. Previous methods rely on output-side signals such as perplexity and consistency checks to detect such attacks. Nevertheless, our analysis reveals that deliberate attacks often induce false confidence, where poisoned outputs exhibit even lower perplexity than benign ones, rendering uncertainty-based detection ineffective. To address this challenge, we explore the internal dynamics of the generator and identify a distinctive signature termed \textit{Attention Collapse}. Unlike the dispersed attention in benign generations, attacked generations exhibit a decrease in entropy as attention concentrates on poisoned documents. Building on these findings, we propose \texttt{D-SCAN} (Document-level Signal Collapse Analysis), a lightweight detection framework that monitors attention dynamics to identify attacked generations. Extensive experiments on multiple attack benchmarks demonstrate the effectiveness of our method. Moreover, D-SCAN can detect attacks even when they fail to alter the final answer. Code is available at https://github.com/yingtaoren/D-Scan.git.

Problem

• RAG poisoning
• adversarial attacks
• attention collapse
• detection
• large language models

Innovation

• Attention Collapse
• RAG Poisoning
• D-SCAN
• Retrieval-Augmented Generation
• Adversarial Detection