DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation

2025-09-07

AI Summary

Retrieval-augmented generation (RAG) systems mitigate hallucination but remain vulnerable to membership inference attacks (MIAs); existing response-dependent attacks suffer from limited efficacy due to neglecting interference from non-member retrieved documents. This paper proposes a differential calibration MIA framework: it introduces controlled query perturbations to model the differential sensitivity of model outputs to member versus non-member retrieved documents, and integrates response differential analysis with contribution separation to achieve high-precision membership discrimination. Crucially, this is the first MIA method to calibrate attacks using sensitivity disparities of retrieved documents under perturbation—breaking from conventional response-only paradigms. Evaluated on Flan-T5, it achieves 97.42% AUC and 94.35% accuracy, outperforming baselines by over 40%. It further demonstrates robust advantages of 10–20% on real-world RAG platforms including Dify and MaxKB.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While Retrieval-Augmented Generation (RAG) effectively reduces hallucinations by integrating external knowledge bases, it introduces vulnerabilities to membership inference attacks (MIAs), particularly in systems handling sensitive data. Existing MIAs targeting RAG's external databases often rely on model responses but ignore the interference of non-member-retrieved documents on RAG outputs, limiting their effectiveness. To address this, we propose DCMI, a differential calibration MIA that mitigates the negative impact of non-member-retrieved documents. Specifically, DCMI leverages the sensitivity gap between member and non-member retrieved documents under query perturbation. It generates perturbed queries for calibration to isolate the contribution of member-retrieved documents while minimizing the interference from non-member-retrieved documents. Experiments under progressively relaxed assumptions show that DCMI consistently outperforms baselines--for example, achieving 97.42% AUC and 94.35% Accuracy against the RAG system with Flan-T5, exceeding the MBA baseline by over 40%. Furthermore, on real-world RAG platforms such as Dify and MaxKB, DCMI maintains a 10%-20% advantage over the baseline. These results highlight significant privacy risks in RAG systems and emphasize the need for stronger protection mechanisms. We appeal to the community's consideration of deeper investigations, like ours, against the data leakage risks in rapidly evolving RAG systems. Our code is available at https://github.com/Xinyu140203/RAG_MIA.

Problem

• Addresses vulnerabilities of RAG systems to membership inference attacks
• Mitigates interference from non-member documents in MIA effectiveness
• Proposes differential calibration to isolate member document contributions

Innovation

• Differential calibration MIA for RAG systems
• Query perturbation to isolate member documents
• Minimizes interference from non-member retrieved documents