1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning

2025-08-11

AI Summary

Large language models (LLMs) deployed in multi-source information interaction scenarios—e.g., processing conference abstracts containing mixed public and private content—risk context-aware privacy leakage. Method: We propose a multi-agent privacy-preserving reasoning framework that decouples privacy-sensitive tasks into subtasks (information extraction, classification, verification), introduces information-flow topology modeling to explicitly characterize leakage paths from upstream errors to downstream components, and employs iterative collaborative verification to mitigate privacy overload. The framework is compatible with both open- and closed-weight LLMs. Results: Evaluated on ConfAIde and PrivacyLens benchmarks, it reduces private information leakage by 18% and 19%, respectively, under GPT-4o, while preserving public-content generation quality—outperforming single-agent baselines. Its core innovation lies in a task-decoupling–driven, controllable information-flow governance mechanism.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Addressing contextual privacy concerns remains challenging in interactive settings where large language models (LLMs) process information from multiple sources (e.g., summarizing meetings with private and public information). We introduce a multi-agent framework that decomposes privacy reasoning into specialized subtasks (extraction, classification), reducing the information load on any single agent while enabling iterative validation and more reliable adherence to contextual privacy norms. To understand how privacy errors emerge and propagate, we conduct a systematic ablation over information-flow topologies, revealing when and why upstream detection mistakes cascade into downstream leakage. Experiments on the ConfAIde and PrivacyLens benchmark with several open-source and closed-sourced LLMs demonstrate that our best multi-agent configuration substantially reduces private information leakage ( extbf{18%} on ConfAIde and extbf{19%} on PrivacyLens with GPT-4o) while preserving the fidelity of public content, outperforming single-agent baselines. These results highlight the promise of principled information-flow design in multi-agent systems for contextual privacy with LLMs.

Problem

• Addressing contextual privacy concerns in multi-source LLM interactions
• Reducing private information leakage via multi-agent reasoning framework
• Analyzing privacy error propagation in information-flow topologies

Innovation

• Multi-agent framework for privacy reasoning
• Systematic ablation over information-flow topologies
• Reduces private information leakage significantly