🤖 AI Summary
This study systematically evaluates the performance of OpenAI’s Privacy Filter (OPF) in detecting personally identifiable information (PII) across diverse languages and domains. Leveraging 32 benchmarks spanning 14 languages and 5 domains, it compares zero-shot OPF against few-shot fine-tuned XLM-RoBERTa. Results show that OPF achieves strong performance on structured PII types such as email addresses and phone numbers (F1: 0.76–0.78) but exhibits significant degradation on non-Latin scripts and culturally specific entities. Notably, fine-tuning XLM-RoBERTa with only 100–1,000 labeled examples consistently surpasses OPF. This work presents the first comprehensive assessment of OPF’s generalization limits, proposes a data- and domain-aware model selection strategy, and demonstrates that binary labeling outperforms fine-grained schemes under few-shot conditions.
📝 Abstract
We present the first independent, systematic evaluation of OpenAI's Privacy Filter (OPF), a 1.5B-parameter bidirectional PII detector, across 42 synthetic benchmarks spanning 22 languages and 5 domains. Zero-shot, OPF achieves F1=0.855 on AI4Privacy and 0.464 on SPY medical, outperforming Presidio (0.431, 0.273) and XLM-RoBERTa (0.269, 0.111) on PII-annotated benchmarks; on multilingual NER, XLM-RoBERTa leads OPF on all 13 Indic and non-Latin languages. GPT-4o leads on medical, legal, and financial PII (SPY: 0.643 avg, Gretel: 0.527), while OPF leads on structured synthetic PII (0.71 avg) and customer support (0.60). OPF degrades sharply when PII is embedded in narrative prose: F1=0.04--0.57 on NER benchmarks and collapse for non-Latin scripts (Arabic: 0.04, Cyrillic: 0.03). Error analysis shows OPF is strongest on structurally regular PII types (email: 0.78, phone: 0.76) and weakest on culturally variable ones (person: 0.40, address: 0.49), and is recall-biased on customer-support and medical/legal PII (P=0.31--0.54, R=0.70--0.85); global precision spans 0.31--0.86 across all domains.