Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models

2024-08-05

arXiv.org

AI Summary

This work identifies and systematically mitigates prompt leakage in customized large language models (LLMs)—a critical threat to service providers’ intellectual property and a potential enabler of downstream attacks. We first establish that prompt leakage arises from two intrinsic mechanisms: (1) the model’s high familiarity with prompt tokens (manifested as low perplexity), and (2) the existence of direct translation paths in the attention matrices. Building on this analysis, we propose a robust defense framework integrating scaling-law analysis, attention visualization, perplexity modeling, and safety-aligned intervention. Evaluated on Llama2-7B and GPT-3.5, our method reduces prompt extraction rates by 83.8% and 71.0%, respectively—substantially outperforming baseline approaches. All code is publicly released to foster reproducibility and further research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Large-scale security measurements

Abstract

The drastic increase of large language models' (LLMs) parameters has led to a new research direction of fine-tuning-free downstream customization by prompts, i.e., task descriptions. While these prompt-based services (e.g. OpenAI's GPTs) play an important role in many businesses, there has emerged growing concerns about the prompt leakage, which undermines the intellectual properties of these services and causes downstream attacks. In this paper, we analyze the underlying mechanism of prompt leakage, which we refer to as prompt memorization, and develop corresponding defending strategies. By exploring the scaling laws in prompt extraction, we analyze key attributes that influence prompt extraction, including model sizes, prompt lengths, as well as the types of prompts. Then we propose two hypotheses that explain how LLMs expose their prompts. The first is attributed to the perplexity, i.e. the familiarity of LLMs to texts, whereas the second is based on the straightforward token translation path in attention matrices. To defend against such threats, we investigate whether alignments can undermine the extraction of prompts. We find that current LLMs, even those with safety alignments like GPT-4, are highly vulnerable to prompt extraction attacks, even under the most straightforward user attacks. Therefore, we put forward several defense strategies with the inspiration of our findings, which achieve 83.8% and 71.0% drop in the prompt extraction rate for Llama2-7B and GPT-3.5, respectively. Source code is avaliable at https://github.com/liangzid/PromptExtractionEval.

Problem

• Analyzing prompt leakage mechanisms in LLMs
• Exploring scaling laws influencing prompt extraction
• Developing strategies to mitigate prompt extraction

Innovation

• Analyzing prompt memorization mechanisms
• Proposing perplexity-based extraction hypotheses
• Developing defense strategies against extraction