R.R.: Unveiling LLM Training Privacy through Recollection and Ranking

2025-02-18

AI Summary

This work reveals a critical privacy vulnerability in large language models (LLMs): even when training data undergoes rigorous PII masking—i.e., all personally identifiable information is scrubbed—LLMs can still be induced to “recall” and reconstruct specific PII entities. To exploit this, we propose R.R., the first two-stage privacy extraction framework: (1) *Recollection*, where semantic prompting guides the target model to complete masked PII tokens; and (2) *Ranking*, where a reference model—leveraging calibration and membership inference–inspired confidence scoring—ranks candidate PII entities. R.R. integrates prompt engineering, PII identifier matching, and re-ranking heuristics. Evaluated on three standard PII benchmarks, it significantly outperforms existing baselines. Crucially, it provides the first empirical evidence refuting the implicit assumption that scrubbed training data precludes PII reconstruction—thereby exposing a substantive privacy gap in current LLM data sanitization practices.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

Large Language Models (LLMs) pose significant privacy risks, potentially leaking training data due to implicit memorization. Existing privacy attacks primarily focus on membership inference attacks (MIAs) or data extraction attacks, but reconstructing specific personally identifiable information (PII) in LLM's training data remains challenging. In this paper, we propose R.R. (Recollect and Rank), a novel two-step privacy stealing attack that enables attackers to reconstruct PII entities from scrubbed training data where the PII entities have been masked. In the first stage, we introduce a prompt paradigm named recollection, which instructs the LLM to repeat a masked text but fill in masks. Then we can use PII identifiers to extract recollected PII candidates. In the second stage, we design a new criterion to score each PII candidate and rank them. Motivated by membership inference, we leverage the reference model as a calibration to our criterion. Experiments across three popular PII datasets demonstrate that the R.R. achieves better PII identical performance compared to baselines. These results highlight the vulnerability of LLMs to PII leakage even when training data has been scrubbed. We release the replicate package of R.R. at a link.

Problem

• Reconstruct PII from scrubbed LLM data
• Introduce recollection and ranking attack
• Highlight LLM vulnerability to PII leakage

Innovation

• Recollect masked PII
• Rank PII candidates
• Use reference model calibration