perform membership inference testing

This work identifies a fundamental flaw in current evaluation methodologies for membership inference (MI) attacks against foundation models: member and non-member samples are typically drawn from disparate distributions, causing standard metrics—such as AUC—to reflect data distribution shift rather than genuine model memorization or privacy leakage. To address this, the authors propose the first model-agnostic “blind baseline” for MI—namely, zero-knowledge classifiers leveraging text statistics or embedding distances—requiring no access to the target model. They systematically evaluate it across eight public MI benchmark datasets. Results show that this blind baseline consistently achieves significantly higher AUC than state-of-the-art MI attacks on all datasets, with remarkable cross-dataset stability. The study demonstrates that prevailing MI evaluation paradigms primarily capture distributional discrepancies—not true membership information leakage—thereby challenging their validity as privacy assessment tools and providing both theoretical grounding and an empirical benchmark for developing more robust privacy evaluation frameworks.

A unified, multimodal benchmark for training data detection (TDD) evaluation is currently lacking. Method: This paper introduces TDDBench—the first cross-modal TDD benchmark—comprising 13 datasets across three modalities (images, tabular data, and text), and systematically evaluates 21 TDD methods under four paradigms: black-box, white-box, gradient-based, and reconstruction-based. Contribution/Results: TDDBench establishes the first standardized, multidimensional evaluation framework, jointly measuring accuracy (accuracy/F1-score), efficiency (inference latency), and resource cost (memory footprint). Experiments reveal that existing TDD methods exhibit limited detection capability, particularly under cross-modal settings and realistic noise conditions. The benchmark platform is fully open-source, modular, and extensible, enabling reproducible algorithm comparison, quantitative analysis, and practical deployment of TDD techniques.

This paper addresses the legal evidentiary challenge of proving training data provenance for foundation models. We argue that membership inference attacks (MIAs) are fundamentally unsuitable for judicial settings due to their inability to construct a valid null hypothesis distribution and guarantee low false-positive rates—critical requirements for legal admissibility. Through rigorous theoretical analysis grounded in statistical hypothesis testing, we formally demonstrate that existing MIAs lack statistical reliability as legally admissible evidence. To overcome this limitation, we propose two verifiable alternatives: (1) a deterministic proof paradigm based on data extraction attacks, and (2) a statistically grounded verification paradigm integrating controlled canary data with enhanced MIAs. Both approaches provably achieve bounded false-positive rates, establishing the first evidence-generation framework for model training provenance that is both statistically rigorous and practically deployable in legal contexts.

To address copyright infringement and transparency concerns arising from unauthorized use of third-party data in machine learning model training, this paper proposes the first general-purpose, task-agnostic data usage auditing framework for black-box models. Methodologically, it innovatively integrates arbitrary black-box membership inference techniques with a custom sequential probability ratio test (SPRT), enabling zero assumptions about downstream tasks, strict control over false positive rates (tunable within 0.5%–5%), and cross-model generalization. The framework features a model-agnostic interface, supporting heterogeneous architectures including image classifiers and multimodal large language models. Extensive experiments on ImageNet classifiers and multimodal foundation models demonstrate an average detection accuracy exceeding 92%, with false positive rates consistently meeting user-specified thresholds. This work significantly enhances the quantifiability and reliability of training data provenance auditing.

Existing membership inference attack (MIA) evaluations average privacy risk across datasets, ignoring individual record-level risk and thereby severely distorting risk estimates for specific models or synthetic data releases. This problem arises from confounding multiple random sources—particularly dataset variability and weight initialization—in current evaluation protocols. We identify this as a fundamental methodological flaw and establish, for the first time, a rigorous evaluation paradigm wherein weight initialization is the sole source of randomness. To address it, we propose a target-data-aware strong adversary model, integrating theoretical risk decomposition, controlled-variable experiments, and state-of-the-art MIA methods. Our empirical analysis reveals that standard evaluations systematically underestimate risk for high-risk samples. In contrast, our framework substantially improves the precision of individual-level privacy risk quantification, and incorporating target-data priors significantly boosts attack success rates.

Pretrained machine learning models may embed novel malicious behaviors that evade detection by static scanning. This work proposes a dynamic, lifecycle-aware security analysis approach that, for the first time, partitions model execution into predictable phases and identifies attacks by monitoring the structured impact of each phase on the host system—without relying on model format or known signatures. Building upon this insight, we develop Re-Moat, a cross-framework runtime monitoring system that achieves full-spectrum detection across all evaluated attack categories with near-zero false positives on a dataset of 77,974 real-world models and multiple proof-of-concept attacks, significantly outperforming existing solutions.

This work addresses the lack of a systematic evaluation framework in existing membership inference attack (MIA) research, which hinders accurate characterization of privacy risks in real-world scenarios. The paper proposes the first end-to-end MIA evaluation framework encompassing data, model architectures, training algorithms, and post-training modules. Under a unified formal threat model, it introduces multidimensional metrics—such as balanced accuracy and true positive rate at low false positive rates—to accommodate both symmetric and asymmetric misclassification costs. Through large-scale empirical analysis across diverse configurations, the study reveals the strong dependence of MIA performance on the choice of threat model and evaluation metrics, leading to practical guidelines for privacy assessment. An open-source, ready-to-use auditing toolkit is released to significantly enhance the reliability and reproducibility of privacy risk evaluations in real-world deployments.

Current evaluations of membership inference attacks (MIAs) against language models suffer from statistical invalidity due to distributional shifts between member and non-member data, hindering fair comparisons. This work proposes an unbiased evaluation benchmark that leverages the temporal in-distribution property observed during model training: by utilizing intermediate checkpoints of open-source large language models (e.g., Pythia, OLMo), it constructs temporally adjacent member and non-member datasets that share the same underlying distribution. We introduce Pandora_LLM, a modular and open-source MIA attack library, and conduct systematic evaluations across models ranging from 70M to 7B parameters. Our experiments reveal the true performance of various MIA methods under this unbiased setting, thereby advancing standardized and reliable assessment of membership privacy risks in language models.

Existing black-box membership inference attacks struggle to effectively identify specific samples from the pretraining data of diffusion models, particularly exhibiting limited discriminative power for low-exposure instances. This work proposes SD-MIA, the first black-box membership inference framework tailored for closed-source platforms, which requires no access to internal model features. Instead, SD-MIA analyzes the model’s denoising responses to a target image paired with perturbed text prompts, establishing a cross-modal collaborative perturbation mechanism to extract highly discriminative membership signals. Experimental results demonstrate that SD-MIA substantially outperforms existing black-box methods on both established benchmarks and a newly curated dataset, achieving performance that even surpasses several white-box baselines and setting a new state of the art in pretraining data membership inference.

This study addresses the critical gap between theory and practice in AI-driven cyberattack prediction, focusing on outdated datasets, limited attack coverage, insufficient model interpretability, weak adversarial robustness, and privacy-ethical risks. Through a systematic review of over 150 benchmark datasets and more than 200 studies, the work introduces a novel multidimensional gap assessment framework based on detection impact, implementation cost, and remediation time to prioritize these challenges. The analysis identifies dataset obsolescence and adversarial robustness as the highest-priority issues, while highlighting interpretability as a cost-effective entry point in resource-constrained settings. Furthermore, the study proposes a tripartite classification of dataset quality—production-ready, research-only, and unusable—alongside a corresponding deployment roadmap, significantly enhancing the practical feasibility and robustness of AI-based cybersecurity systems.