multi-stage pii detection

To address performance bottlenecks in PII detection for low-resource languages—caused by scarce annotated data and linguistic diversity—this paper proposes RECAP, a hybrid framework integrating deterministic regular expressions with context-aware large language models (LLMs) within a modular, three-stage refinement pipeline. RECAP enables zero-shot generalization across 13 languages and 300+ entity types without retraining for new categories. It first applies regex-based coarse filtering, followed by LLM-driven fine-grained recognition, and finally rule-guided disambiguation and boundary refinement. This design significantly enhances boundary detection and ambiguity resolution. On the nervaluate benchmark, RECAP achieves a weighted F1-score of 89.7%, outperforming fine-tuned NER models by 82% and zero-shot LLM baselines by 17%. The framework delivers an efficient, scalable solution for multilingual privacy compliance.

This work addresses the privacy leakage risk of personally identifiable information (PII) embedded in user prompts for large language models (LLMs). We propose the first query-aware PII privacy protection evaluation framework. Methodologically, we design a query-agnostic PII masking strategy and construct a fine-grained (55 categories), multi-scenario (single- and multi-subject interaction) standardized benchmark; it is built upon 2,842 manually curated samples and integrates contextual modeling, query intent alignment, and ground-truth answer annotation to enable end-to-end assessment. Our key contribution is the first deep coupling of PII masking with query relevance judgment—revealing that mainstream LLMs exhibit severe deficiencies in relevance identification within multi-subject interactions, thereby identifying intelligent selective masking as a critical bottleneck in practical PII protection.

Detecting heterogeneous, context-sensitive personally identifiable information (PII) across multiple languages and scenarios faces dual challenges of scarce labeled data and privacy risks. This work proposes a lightweight solution by constructing a 0.3B-parameter model based on the GLiNER2 architecture, trained via character-level span annotation and transfer learning. The approach introduces a constraint-driven generation pipeline to synthesize 4,910 multilingual text samples, marking the first integration of constraint-driven synthetic data with an efficient named entity recognition framework to enable accurate cross-lingual extraction of 42 PII categories. Evaluated on the SPY benchmark, the method achieves significantly higher span-level F1 scores than five existing systems, including the OpenAI Privacy Filter, and the model is publicly released to advance open research in PII detection.

Large language models (LLMs) pose privacy risks through extraction of personally identifiable information (PII) memorized from training data; however, existing evaluation methods—relying on generic, context-agnostic prompts—severely underestimate real-world attack success rates. To address this, we propose *domain-semantic anchoring*: grounding PII extraction prompts with domain-specific data to enhance their semantic relevance and effectiveness. Our empirical study is the first to demonstrate that this method increases PII extraction success by over an order of magnitude. We further introduce a red-teaming evaluation paradigm that better approximates realistic adversarial behavior, substantially correcting prior underestimations of privacy risk. In experiments, single-query PII extraction reaches 0.92%; success rises to 3.9% after 128 queries and 6.86% after 2,308 queries—equivalent to successfully extracting PII from approximately 1 in 15 individuals. This work establishes a more credible quantitative benchmark for LLM data memorization privacy risks and informs practical mitigation strategies.

This work addresses the limitations of existing PII detection benchmarks, which suffer from narrow entity coverage and uncontrolled generation conditions that obscure failure mechanisms of detectors. We present the first systematically controlled multilingual PII detection benchmark, spanning 25 languages, 51 entity types, and 4,127 surface form patterns. Generation is governed by a strength-2 covering array sampler that modulates nine dimensions, complemented by a GDPR-aligned sensitivity stratification mechanism. Our approach innovatively integrates multilingual synthetic data generation, entity-level metadata annotation, LLM-as-judge evaluation, and hierarchical test set construction. Evaluation reveals that rule-based systems exhibit alarmingly low recall—dropping to 0.07—for high-sensitivity categories, whereas large language models demonstrate greater robustness; sensitivity stratification emerges as the most challenging dimension. The full benchmark and associated tools are publicly released.

This work addresses the limited generalization and poor adaptability of privacy auditing models caused by scarce annotated data and rigid, predefined personally identifiable information (PII) taxonomies. To overcome these challenges, the authors propose a multi-stage large language model (LLM) pipeline that enables explicit PII value annotation under arbitrary PII classification schemes at runtime. The approach integrates deterministic preprocessing, label-level classification, instance-level annotation, and output validation, achieving taxonomy-agnostic dynamic PII labeling for the first time. Additionally, they introduce an LLM-based synthetic HTTP traffic generation technique that facilitates controlled evaluation and prompt engineering without relying on real sensitive data. Experiments across three diverse PII taxonomies—varying in domain and granularity—demonstrate the method’s effectiveness in accurately identifying PII types and extracting corresponding values, highlighting the potential of LLMs for flexible privacy annotation and synthetic data generation.

This study systematically evaluates the performance of OpenAI’s Privacy Filter (OPF) in detecting personally identifiable information (PII) across diverse languages and domains. Leveraging 32 benchmarks spanning 14 languages and 5 domains, it compares zero-shot OPF against few-shot fine-tuned XLM-RoBERTa. Results show that OPF achieves strong performance on structured PII types such as email addresses and phone numbers (F1: 0.76–0.78) but exhibits significant degradation on non-Latin scripts and culturally specific entities. Notably, fine-tuning XLM-RoBERTa with only 100–1,000 labeled examples consistently surpasses OPF. This work presents the first comprehensive assessment of OPF’s generalization limits, proposes a data- and domain-aware model selection strategy, and demonstrates that binary labeling outperforms fine-grained schemes under few-shot conditions.

This study addresses the limited generalization of existing personally identifiable information (PII) detection systems, which often suffer from narrow training data domains and struggle to achieve broad coverage across heterogeneous text. Leveraging a refined multi-source PIIBench dataset, the authors systematically evaluate three DeBERTa fine-tuning strategies—direct fine-tuning, source-conditional hierarchical modeling (SC+H), and a three-stage curriculum learning approach (SC+H+Curr)—across 82 fine-grained PII entity types. Results demonstrate that direct fine-tuning, using only diverse task data and a simple weighted cross-entropy loss, achieves F1 scores of 0.6476 and 0.6455 on the test_5k and full 100k test sets, respectively, significantly outperforming current methods. This strategy leads in 54 fine-grained and all 10 coarse-grained categories, underscoring the critical role of data diversity and a concise objective function in enhancing model generalization.