Learning to Erase Private Knowledge from Multi-Documents for Retrieval-Augmented Large Language Models

2025-04-14

AI Summary

To address privacy leakage arising from multi-document retrieval in Retrieval-Augmented Generation (RAG), this paper introduces, for the first time, the customizable private knowledge erasure task—aiming to precisely remove user-specified sensitive information while preserving public knowledge. Methodologically: (1) a global knowledge graph is constructed to resist de-anonymization attacks; (2) a multi-document knowledge segmentation and rewriting mechanism is designed to isolate and transform private content; and (3) private knowledge removal is jointly optimized with factual consistency via end-to-end training—combining Flan-T5 fine-tuning and Proximal Policy Optimization (PPO)-based reinforcement learning. Experiments across four QA benchmarks demonstrate that our approach significantly outperforms GPT-4o in erasure efficacy, substantially reduces privacy leakage risk, and maintains stable generation quality and factual accuracy.

Technology Category

• Machine Learning: Privacy
• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-Augmented Generation (RAG) is a promising technique for applying LLMs to proprietary domains. However, retrieved documents may contain sensitive knowledge, posing risks of privacy leakage in generative results. Thus, effectively erasing private information from retrieved documents is a key challenge for RAG. Unlike traditional text anonymization, RAG should consider: (1) the inherent multi-document reasoning may face de-anonymization attacks; (2) private knowledge varies by scenarios, so users should be allowed to customize which information to erase; (3) preserving sufficient publicly available knowledge for generation tasks. This paper introduces the privacy erasure task for RAG and proposes Eraser4RAG, a private knowledge eraser which effectively removes user-defined private knowledge from documents while preserving sufficient public knowledge for generation. Specifically, we first construct a global knowledge graph to identify potential knowledge across documents, aiming to defend against de-anonymization attacks. Then we randomly split it into private and public sub-graphs, and fine-tune Flan-T5 to rewrite the retrieved documents excluding private triples. Finally, PPO algorithm optimizes the rewriting model to minimize private triples and maximize public triples retention. Experiments on four QA datasets demonstrate that Eraser4RAG achieves superior erase performance than GPT-4o.

Problem

• Erase private knowledge from retrieved documents in RAG
• Defend against de-anonymization attacks in multi-document reasoning
• Balance private erasure and public knowledge retention

Innovation

• Constructs global knowledge graph for privacy defense
• Splits knowledge into private and public sub-graphs
• Optimizes rewriting model with PPO algorithm