🤖 AI Summary
This paper reveals a dual privacy risk of code-generation large language models (LLMs) stemming from training data memorization: *inadvertent leakage*—where sensitive information (e.g., API keys) is exposed without user prompting—and *malicious leakage*—where adversaries exploit prior knowledge to induce such disclosures. We propose the first decoupled, parallel evaluation framework integrating data-mining–based privacy testing, a customized secret injection/detection benchmark, and cross-version comparative analysis. Systematic evaluation on the OLMo model family and Dolma dataset demonstrates that: (1) data preprocessing adjustments exert non-monotonic effects—simultaneously increasing inadvertent leakage probability while decreasing malicious leakage probability; and (2) leakage susceptibility depends critically on sensitive information type (e.g., API keys, passwords, internal paths), not merely prompt engineering. Our work establishes a new paradigm for privacy risk assessment and data governance in code LLMs.
📝 Abstract
This paper explores the risk that a large language model (LLM) trained for code generation on data mined from software repositories will generate content that discloses sensitive information included in its training data. We decompose this risk, known in the literature as ``unintended memorization,'' into two components: unintentional disclosure (where an LLM presents secrets to users without the user seeking them out) and malicious disclosure (where an LLM presents secrets to an attacker equipped with partial knowledge of the training data). We observe that while existing work mostly anticipates malicious disclosure, unintentional disclosure is also a concern. We describe methods to assess unintentional and malicious disclosure risks side-by-side across different releases of training datasets and models. We demonstrate these methods through an independent assessment of the Open Language Model (OLMo) family of models and its Dolma training datasets. Our results show, first, that changes in data source and processing are associated with substantial changes in unintended memorization risk; second, that the same set of operational changes may increase one risk while mitigating another; and, third, that the risk of disclosing sensitive information varies not only by prompt strategies or test datasets but also by the types of sensitive information. These contributions rely on data mining to enable greater privacy and security testing required for the LLM training data supply chain.