Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models

2026-06-08

AI Summary

This work addresses the unclear practical privacy guarantees of differential privacy (DP) in large language model adaptation, particularly when adaptation data overlaps with or is correlated to pretraining data. The authors propose a comprehensive privacy evaluation framework spanning the entire pretraining-to-adaptation pipeline and conduct systematic benchmarking to assess the robustness of DP-adapted models against membership inference and canary extraction attacks under varying data distributions. They reveal, for the first time, that distributional shift critically influences empirical privacy risk: adaptation data closer to the pretraining distribution— even without direct overlap—leads to higher privacy leakage. Moreover, parameter-efficient fine-tuning methods such as LoRA demonstrate substantially stronger empirical privacy than full fine-tuning in out-of-distribution settings, offering actionable guidance for deploying models in privacy-sensitive applications.

Abstract

Recent work has applied differential privacy (DP) to adapt large language models (LLMs) for sensitive applications, offering theoretical guarantees. However, its practical effectiveness remains unclear, partly due to LLM pretraining, where overlaps and interdependencies with adaptation data can undermine privacy despite DP efforts. To analyze this issue in practice, we investigate privacy risks under DP adaptations in LLMs using state-of-the-art attacks such as robust membership inference and canary data extraction. We benchmark these risks by systematically varying the adaptation data distribution, from exact overlaps with pretraining data, through in-distribution (IID) cases, to entirely out-of-distribution (OOD) examples. Additionally, we evaluate how different adaptation methods and different privacy regimes impact the vulnerability. Our results show that distribution shifts strongly influence privacy vulnerability: the closer the adaptation data is to the pretraining distribution, the higher the practical privacy risk at the same theoretical guarantee, even without direct data overlap. We find that parameter-efficient fine-tuning methods, such as LoRA, achieve the highest empirical privacy protection for OOD data. Our benchmark identifies key factors for achieving practical privacy in DP LLM adaptation, providing actionable insights for deploying customized models in sensitive settings. Looking forward, we propose a structured framework for holistic privacy assessment beyond adaptation privacy, to identify and evaluate risks across the full pretrain-adapt pipeline of LLMs.

Problem

• differential privacy
• large language models
• privacy risk
• data distribution shift
• empirical privacy

Innovation

• differential privacy
• large language models
• distribution shift
• parameter-efficient fine-tuning
• empirical privacy evaluation