Can Prompt Anonymity Protect Your Identity From LLM Providers?

2026-09-27

AI Summary

This study investigates whether anonymizing large language model (LLM) prompts can effectively prevent user re-identification. To this end, it presents the first empirical investigation into prompt author re-identification by constructing PromptAnonBench, a large-scale, real-world privacy benchmark comprising multi-turn dialogues. The proposed approach is systematically evaluated using embedding vector analysis and text-based privacy defense techniques. Experimental results demonstrate that 50% to 75% of specific users can be successfully re-identified even under a stringent 10% false positive rate. These findings reveal critical privacy vulnerabilities inherent in relying solely on prompt anonymization, thereby providing an essential benchmark and robust empirical evidence for advancing data privacy protection mechanisms in large language models.

Abstract

User conversations with large language models (LLMs) often contain highly sensitive personal information that can be exploited by LLM providers to create detailed user dossiers, enable targeted advertising, and train more powerful models. To protect user privacy, anonymizing LLM proxies have emerged as a practical solution that separates user identity from their prompts, yet this approach still leaves the prompt content visible to LLM providers. We study the impact of this gap by conducting the first empirical investigation into the risk of prompt authorship re-identification. Towards this end, we create PromptAnonBench, a novel benchmark for evaluating prompt anonymity, consisting of over 175,000 cleaned, authentic multi-turn user prompts from various real-world datasets (SWE-Chat and WildChat). Using the embeddings of historical user conversations, an attacker can correctly detect and re-identify at least one anonymized conversation for 50--75% of SWE-Chat users and up to 10% of WildChat users at a 10% false acceptance rate for out-of-set users, even with text-based defenses applied. Our findings unveil the risk of relying only on anonymity for private LLM inference and the gap in existing text privacy defenses.

Problem

• Prompt Anonymity
• Authorship Re-identification
• Large Language Models
• User Privacy
• Text Privacy Defenses

Innovation

• Prompt Anonymity
• Authorship Re-identification
• PromptAnonBench
• LLM Privacy
• Text Embeddings