Near-Duplicate Families Break Exact-Record Membership Inference

2026-09-27

AI Summary

This study addresses the fundamental confounding in copyright auditing caused by near-duplicates prevalent in web data, which render conventional membership inference attacks incapable of distinguishing exact records from similar data families. To resolve this, we introduce a novel "four-world" auditing framework that leverages the LiRA algorithm and controlled intervention experiments on the CC-News dataset to effectively decouple the effects of exact record inclusion from family presence. Our findings demonstrate that natural near-duplicate families induce severe misattribution, revealing that families can substitute for exact records in classification tasks while autoregressive models retain only residual signals. Consequently, we establish that positive evidence can solely confirm family-level exposure. This work fundamentally challenges the core assumptions underlying existing data attribution methodologies.

Abstract

Membership inference (MI) asks whether a specific record appeared in a model's training set and is increasingly used as evidence for data provenance and copyright auditing. These applications require determining whether the exact queried record was used for training, rather than merely whether the model was exposed to similar content. Making this distinction is challenging because web-scale datasets naturally contain near-duplicates, including syndicated articles, mirrored pages, and lightly modified images. We show that this creates a fundamental confound for standard MI. A clean-reference audit typically calibrates membership against a null in which neither the queried record nor its near-duplicate family is present. In deployment, however, the queried record may be absent while a non-identical family member was used for training. We introduce a four-world audit that independently varies exact-record inclusion and family presence to separate these cases. Natural near-duplicate families cause severe false attribution. On CC-News, a clean-reference LiRA auditor labels 99.70% of family-present exact non-members as members at 1.00% false-positive rate. This failure persists across alternative scores, model architectures, and executed deduplication and retraining. Controlled interventions further reveal that the effect depends on the learning objective. In classification, faithful families largely substitute for the exact record, reducing exact-given-family inference to near chance. In autoregressive language modeling, the exact sequence retains a detectable residual, while family presence still confounds clean-reference decisions. These results show that positive model-only membership evidence may establish family-level exposure without establishing exact-record provenance.

Problem

• Membership Inference
• Near-Duplicate Families
• Exact-Record Provenance
• False Attribution
• Data Auditing

Innovation

• Membership Inference
• Near-Duplicate Families
• Four-World Audit
• Data Provenance
• LiRA