Defense Against Prompt Inversion Attacks: An Information-Theoretic Approach for LLM Collaborative Inference

2026-06-09

AI Summary

This work addresses the vulnerability of intermediate activations to prompt inversion attacks in edge-cloud collaborative inference, which can lead to reconstruction of users’ sensitive inputs. To mitigate this risk, the authors propose an information-theoretic privacy-preserving framework that learns representations minimizing mutual information between intermediate features and the original input, while satisfying task utility and latency constraints. The study establishes the first theoretical privacy guarantees against prompt inversion attacks, characterizes the fundamental trade-off between privacy and utility, and introduces a low-dimensional information bottleneck-based privacy adapter mechanism. Experimental results across multiple scenarios demonstrate that the proposed method significantly outperforms existing defenses, reducing attack success rates by up to 35% and achieving superior overall performance in balancing privacy, utility, and latency.

Abstract

Collaborative edge-cloud inference enables resource-constrained devices to leverage large language models (LLMs) by offloading partial computation to cloud servers. However, transmitting intermediate activations exposes sensitive user prompts to prompt inversion attacks, where an adversary reconstructs the original input from shared representations. Existing defenses rely largely on heuristic perturbations or empirical tuning, offering limited theoretical understanding of privacy leakage and its interaction with utility and latency constraints. We propose an information-theoretic defense framework for prompt inversion in collaborative LLM inference. Our approach learns privacy-preserving representations by explicitly minimizing the mutual information between intermediate activations and the input prompt while maintaining task utility under computational constraints. We derive theoretical guarantees on prompt reconstruction error, characterize fundamental privacy-utility tradeoffs, and establish token-level accuracy bounds for downstream inference. We then propose a novel defense based on privacy adapters implemented via low-dimensional information bottlenecks. Extensive experiments across multiple settings demonstrate that our method achieves superior privacy-utility-latency tradeoffs compared to existing defenses (up to 35% reduction in attack success), providing a principled foundation for private and efficient collaborative LLM inference.

Problem

• prompt inversion attacks
• collaborative inference
• privacy leakage
• large language models
• intermediate activations

Innovation

• information-theoretic privacy
• prompt inversion defense
• collaborative LLM inference
• privacy-utility tradeoff
• information bottleneck