Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention

2026-02-04

AI Summary

This work addresses the vulnerability of Retrieval-Augmented Generation (RAG) systems to corpus poisoning attacks, wherein adversaries inject misleading documents to manipulate large language model outputs. To counter this threat, the authors propose Sparse Document Attention for Guarding (SDAG), which introduces block-sparse attention into RAG defense for the first time. By modifying only the attention mask during inference, SDAG effectively blocks cross-attention among retrieved documents, thereby preventing inter-document contamination. Notably, this approach requires neither model fine-tuning nor architectural changes and is compatible with existing defense strategies. Experimental results demonstrate that SDAG significantly reduces attack success rates across diverse poisoning scenarios, and its combination with state-of-the-art defenses yields statistically significant performance improvements.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval Augmented Generation (RAG) is a highly effective paradigm for keeping LLM-based responses up-to-date and reducing the likelihood of hallucinations. Yet, RAG was recently shown to be quite vulnerable to corpus knowledge poisoning: an attacker injects misleading documents to the corpus to steer an LLM's output to an undesired response. We argue that the standard causal attention mechanism in LLMs enables harmful cross-document interactions, specifically in cases of attacks. Accordingly, we introduce a novel defense approach for RAG: Sparse Document Attention RAG (SDAG). This is a block-sparse attention mechanism that disallows cross-attention between retrieved documents. SDAG requires a minimal inference-time change to the attention mask; furthermore, no fine-tuning or additional architectural changes are needed. We present an empirical evaluation of LLM-based question answering (QA) with a variety of attack strategies on RAG. We show that our SDAG method substantially outperforms the standard causal attention mechanism in terms of attack success rate. We further demonstrate the clear merits of integrating SDAG with state-of-the-art RAG defense methods. Specifically, the integration results in performance that is statistically significantly better than the state-of-the-art.

Problem

• Corpus Knowledge Poisoning
• Retrieval Augmented Generation
• RAG Security
• LLM Hallucination
• Adversarial Attacks

Innovation

• Sparse Attention
• RAG
• Knowledge Poisoning
• Cross-Document Interaction
• Defense Mechanism