pii detection

This work addresses the pervasive risk of inadvertent personal identifiable information (PII) leakage by ordinary users during interactions with cloud-based large language models, stemming from a lack of effective control over their sensitive data. To mitigate this, we propose a lightweight browser overlay tool that, for the first time, adapts enterprise-grade PII redaction techniques into a consumer-oriented, client-side solution. The system integrates front-end entity recognition with anonymization mechanisms and introduces an innovative “smokescreen” behavior—generating deceptive interaction patterns to actively disrupt third-party user profiling. Fully open-source and extensible, our approach preserves the usability of AI interactions while significantly enhancing users’ autonomy over their PII, thereby offering robust protection against web-based privacy breaches.

This work addresses the privacy leakage risk of personally identifiable information (PII) embedded in user prompts for large language models (LLMs). We propose the first query-aware PII privacy protection evaluation framework. Methodologically, we design a query-agnostic PII masking strategy and construct a fine-grained (55 categories), multi-scenario (single- and multi-subject interaction) standardized benchmark; it is built upon 2,842 manually curated samples and integrates contextual modeling, query intent alignment, and ground-truth answer annotation to enable end-to-end assessment. Our key contribution is the first deep coupling of PII masking with query relevance judgment—revealing that mainstream LLMs exhibit severe deficiencies in relevance identification within multi-subject interactions, thereby identifying intelligent selective masking as a critical bottleneck in practical PII protection.

Commercial PII deletion services (e.g., DeleteMe, Incogni) claim to remove users’ personal information from data broker databases, yet their efficacy remains unverified by independent empirical evaluation. Method: This work introduces the first large-scale, user-driven evaluation framework—integrating real subscriptions, manual annotation, web scraping for ground-truth comparison, and textual analysis of service claims—to assess coverage, PII identification accuracy, and deletion effectiveness across major services. Results: Only 41.1% of records flagged as PII by services corresponded to users’ actual identities; among verified true PII records, only 48.2% were successfully removed; and all services covered far fewer data brokers than advertised. The study uncovers systemic deficiencies across three dimensions—coverage breadth, classification precision, and operational efficacy—and establishes a reproducible methodology for evaluating privacy-enhancing technologies.

Large language models (LLMs) pose privacy risks through extraction of personally identifiable information (PII) memorized from training data; however, existing evaluation methods—relying on generic, context-agnostic prompts—severely underestimate real-world attack success rates. To address this, we propose *domain-semantic anchoring*: grounding PII extraction prompts with domain-specific data to enhance their semantic relevance and effectiveness. Our empirical study is the first to demonstrate that this method increases PII extraction success by over an order of magnitude. We further introduce a red-teaming evaluation paradigm that better approximates realistic adversarial behavior, substantially correcting prior underestimations of privacy risk. In experiments, single-query PII extraction reaches 0.92%; success rises to 3.9% after 128 queries and 6.86% after 2,308 queries—equivalent to successfully extracting PII from approximately 1 in 15 individuals. This work establishes a more credible quantitative benchmark for LLM data memorization privacy risks and informs practical mitigation strategies.

This work addresses the limited generalization and poor adaptability of privacy auditing models caused by scarce annotated data and rigid, predefined personally identifiable information (PII) taxonomies. To overcome these challenges, the authors propose a multi-stage large language model (LLM) pipeline that enables explicit PII value annotation under arbitrary PII classification schemes at runtime. The approach integrates deterministic preprocessing, label-level classification, instance-level annotation, and output validation, achieving taxonomy-agnostic dynamic PII labeling for the first time. Additionally, they introduce an LLM-based synthetic HTTP traffic generation technique that facilitates controlled evaluation and prompt engineering without relying on real sensitive data. Experiments across three diverse PII taxonomies—varying in domain and granularity—demonstrate the method’s effectiveness in accurately identifying PII types and extracting corresponding values, highlighting the potential of LLMs for flexible privacy annotation and synthetic data generation.

Detecting heterogeneous, context-sensitive personally identifiable information (PII) across multiple languages and scenarios faces dual challenges of scarce labeled data and privacy risks. This work proposes a lightweight solution by constructing a 0.3B-parameter model based on the GLiNER2 architecture, trained via character-level span annotation and transfer learning. The approach introduces a constraint-driven generation pipeline to synthesize 4,910 multilingual text samples, marking the first integration of constraint-driven synthetic data with an efficient named entity recognition framework to enable accurate cross-lingual extraction of 42 PII categories. Evaluated on the SPY benchmark, the method achieves significantly higher span-level F1 scores than five existing systems, including the OpenAI Privacy Filter, and the model is publicly released to advance open research in PII detection.

This work addresses the limitations of existing PII detection benchmarks, which suffer from narrow entity coverage and uncontrolled generation conditions that obscure failure mechanisms of detectors. We present the first systematically controlled multilingual PII detection benchmark, spanning 25 languages, 51 entity types, and 4,127 surface form patterns. Generation is governed by a strength-2 covering array sampler that modulates nine dimensions, complemented by a GDPR-aligned sensitivity stratification mechanism. Our approach innovatively integrates multilingual synthetic data generation, entity-level metadata annotation, LLM-as-judge evaluation, and hierarchical test set construction. Evaluation reveals that rule-based systems exhibit alarmingly low recall—dropping to 0.07—for high-sensitivity categories, whereas large language models demonstrate greater robustness; sensitivity stratification emerges as the most challenging dimension. The full benchmark and associated tools are publicly released.

This work addresses the risk of visual personally identifiable information (PII) leakage when AI agents process web screenshots—a vulnerability overlooked by existing research due to the absence of dedicated benchmarks. To bridge this gap, we introduce WebPII, the first fine-grained, agent-oriented benchmark for synthetic visual PII detection, comprising 44,865 annotated e-commerce interface images. WebPII features an expanded PII taxonomy and forward-looking detection categories such as transaction identifiers and partially filled forms. Leveraging vision-language models to synthesize UIs and auto-annotate PII, we develop WebRedact, a real-time PII recognition and redaction model that achieves a mAP@50 of 0.753—substantially outperforming text-based baselines (0.357)—and runs at 20 ms inference latency on CPU. Both the dataset and model are publicly released.