Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study

2026-08-04

AI Summary

This study investigates whether substituting protected health information (PHI) with same-type surrogates in structure-preserving de-identification affects the ability of downstream PHI detectors to recognize masked spans. To address this, the authors propose a TOST-based equivalence evaluation framework, design a paired multi-detector protocol to disentangle coverage from utility using only masked text, and establish a surrogate failure taxonomy. Experiments spanning 11 detectors, 7 benchmarks, and 7 languages reveal that recall decreases marginally from 76.1% to 74.9%, demonstrating statistical equivalence within a ±2-point margin. Performance degradation is primarily attributed to poorly generated or out-of-distribution surrogates. This work provides the first evidence, across multiple languages, detectors, and benchmarks, that surrogate replacement has negligible impact on PHI detection performance.

Abstract

Structure-preserving de-identification replaces protected health information (PHI) with realistic same-type surrogates -- "Anna S." becomes "Maria S.", not [NAME] -- so that clinical text stays fluent and downstream tools keep working. But this only helps if the substitution does not itself corrupt the signal those tools rely on. We ask a narrow, testable question: on the spans a de-identifier actually masks, can downstream PHI detectors still find the surrogate? We introduce a paired, multi-detector evaluation protocol that (i) scores utility only on masked spans, decoupling coverage from utility; (ii) uses equivalence testing (TOST) rather than null-hypothesis significance testing, which is uninformative at our sample size (57k paired spans); and (iii) builds a surrogate-failure typology separating fixable generator defects from intrinsic detector limits. Across 11 detectors, 7 benchmarks, and 7 languages (1,750 documents), recall on masked spans moves from 76.1% to 74.9% -- a change our equivalence test shows is statistically equivalent to zero within a +/-2-point margin (p ~ 3e-9), with detector ranking preserved. The residual loss does not reflect detectors getting worse at PHI: it concentrates in malformed and out-of-distribution surrogates (truncation Chicago -> Illino, salience loss Cedars-Sinai -> Vidant). A redaction floor and an open-source surrogate baseline indicate the effect is a property of well-formed substitution, not of one tool. We release the evaluation subsets, scoring code, and an interactive dashboard at https://custodianai.pages.dev so the protocol can audit any structure-preserving transform.

Problem

• PHI detection
• surrogate substitution
• de-identification
• structure-preserving
• detectability

Innovation

• structure-preserving de-identification
• surrogate substitution
• equivalence testing
• PHI detectability
• multi-detector evaluation