detection performance evaluation

This study addresses the widespread issue in AI red-teaming evaluations where comparisons of attack success rates (ASR) often rely on invalid or incomparable measurements, leading to erroneous conclusions about system security or attack efficacy. For the first time, the paper introduces measurement validity theory from social sciences into the AI red-teaming domain, systematically analyzing—through inferential statistics and illustrative case studies such as jailbreaking attacks—the conditions under which ASR comparisons are meaningful. The work establishes clear prerequisites for valid ASR comparisons, identifies and categorizes common patterns of invalid comparison, and thereby provides a rigorous theoretical foundation that significantly enhances the scientific rigor and comparability of AI safety evaluations.

This study addresses the limitations of traditional industrial control system (ICS) intrusion detection methods that rely on binary labels, which fail to capture the rich diversity of attack behaviors. The authors propose a physics-informed behavioral representation framework that maps multivariate process trajectories into five interpretable behavioral patterns: drift, spike, oscillation, repetition, and switching, thereby transcending the binary evaluation paradigm. For the first time, cross-dataset hierarchical behavioral evaluation is conducted across multiple ICS benchmarks—including SWaT, WADI, and HAI—exposing significant dataset biases and model blind spots. Experimental results reveal a substantial performance degradation under behavioral stratification (e.g., macro F1 on SWaT drops from 85.44% to 37.84%), highlighting the inadequacy of conventional evaluation metrics in reflecting real-world detection capabilities.

This paper identifies a novel “poisoning” threat to large language models (LLMs) in security automation: even minimal fine-tuning of Llama3.1-8B and Qwen3-4B with poisoned data induces strong, targeted bias, causing LLM-based alert investigation systems to systematically overlook genuine threat alerts from specific sources. We propose the first user-level targeted poisoning attack method and validate its stable evasion of alert review mechanisms via prompt engineering. Experiments demonstrate that such bias fully compromises model performance on critical threat assessment tasks. To mitigate this risk, we introduce three complementary defense strategies: (1) data sanitization, (2) bias detection heuristics, and (3) robust prompt design. Empirical evaluation shows these approaches significantly enhance the trustworthiness and deployment robustness of LLMs in security-critical applications. Our work underscores the vulnerability of production-grade LLMs to subtle data poisoning in high-stakes cybersecurity contexts and provides actionable, empirically validated countermeasures.

This paper studies the targeted robustness evaluation of linear classifiers under label poisoning attacks: given a training set, how to automatically quantify its resilience against attacks that perturb only a few labels to flip the prediction of a specific test instance. The authors first prove that exact computation of this robustness is NP-complete under the constraints of label-only perturbations and limited attacker prior knowledge. To address this intractability, they propose an efficient method based on adversarial label perturbation modeling and optimization, yielding theoretically sound and computationally feasible upper and lower bounds on robustness under hypothesis-space constraints. Experiments on multiple public benchmarks demonstrate the tightness and practical utility of the bounds—exceeding them consistently induces significant performance degradation. Compared to state-of-the-art approaches, the method is broadly applicable, requires no model retraining, and provides the first scalable, automated framework for dataset security assessment.

Existing behavioral malware detection research heavily relies on sandbox-derived features, leading to severe performance degradation—accuracy drops to 20%–50%, far below the reported >90%—when deployed on real endpoints. This exposes three fundamental challenges: label noise, distribution shift, and spurious feature reliance. To address this, we conduct the first large-scale empirical evaluation on real endpoints and propose a robust end-to-end training framework tailored for endpoint environments. Our approach integrates behavioral trajectory modeling, cross-environment distribution alignment, noise-robust learning, and telemetry-driven training leveraging real-world endpoint telemetry. Experiments demonstrate a 5%–30% relative improvement in detection accuracy over sandbox-trained baselines. Crucially, our work shifts the detection paradigm from sandbox-centric training to direct training on endpoint data. As part of this contribution, we release the first publicly available benchmark dataset comprising authentic endpoint behavioral trajectories, enabling reproducible, realistic evaluation of behavioral malware detection systems.

This work addresses the lack of systematic evaluation benchmarks for large language models (LLMs) in security audit log investigation tasks by introducing AuditBench, the first audit log benchmark specifically designed for attack investigation. AuditBench encompasses over 50 real-world scenarios across Linux and Windows systems and focuses on four core tasks: alert classification, persistence mechanism identification, among others. Through multidimensional experiments, the study systematically evaluates the impact of model scale, log representation, prompt design, and fine-tuning strategies on performance and error patterns, while also analyzing the quality of LLM-generated explanations. The findings reveal the capability boundaries and characteristic failure modes of various models across different investigative tasks, providing empirical foundations for deploying and optimizing LLMs in security operations.

This study addresses the challenge of alarm fatigue in continuous model monitoring caused by high false positive rates of existing drift detectors, which undermines monitoring reliability. It presents the first systematic evaluation of the cumulative false positive behavior of five widely used methods—Population Stability Index (PSI), Kolmogorov–Smirnov (KS) test, Maximum Mean Discrepancy (MMD), Least-Squares Density Difference (LSDD), and adversarial validation—under continuous monitoring settings, incorporating Bonferroni correction for multiple hypothesis testing. The empirical analysis reveals that PSI exhibits markedly improved stability when sample sizes exceed 200, whereas KS, MMD, and LSDD demonstrate greater reliability with smaller batch sizes. While Bonferroni correction effectively suppresses false positives, it concurrently reduces detection sensitivity. These findings offer practical guidance for selecting batch sizes and calibrating detectors in real-world deployments, balancing robustness and responsiveness.

This study addresses the critical issue that existing selective prediction methods in signal domains—such as anomalous sound detection and AI-generated image forensics—often yield a false sense of security due to the use of uncalibrated thresholds, resulting in actual error rates that substantially exceed users’ prescribed risk budgets. The work presents the first systematic audit of four distribution-free calibration rules (NAIVE, Hoeffding, Clopper–Pearson, and Betting) regarding their risk control performance on both real and synthetic data. Findings reveal that NAIVE exceeds the risk budget in 49–73% of experiments; Clopper–Pearson and Betting achieve zero violations under exchangeability but suffer 9–30% violation rates when deployed in grouped settings where exchangeability fails. Group-wise thresholding restores valid risk control at the cost of reduced coverage. The study underscores the pivotal role of tight confidence bounds for effective coverage and identifies uncalibrated thresholds as the root cause of risk miscontrol.

Existing prompt injection detectors frequently miss high-severity attacks under distribution shifts while still outputting overconfident predictions near 1.0, a risk substantially underestimated by standard calibration metrics. This work introduces a severity-aware calibration perspective, proposing a severity metric S to quantify the confidence assigned to missed attacks. Through fixed-threshold cross-distribution evaluation, black-box rewrite-based attack generation, and instruction-tuned models as judges, the study systematically investigates detector failure mechanisms. Findings reveal that content keywords—not injection syntax—are the primary cause of detection blind spots. Critically, all evaluated detectors exhibit highly confident false negatives across diverse distribution shifts, exposing fundamental limitations in current calibration approaches.

This work addresses the vulnerability of Retrieval-Augmented Generation (RAG) systems to polymorphic Sybil poisoning attacks—coordinated, diverse misinformation injections—and introduces the first evaluation framework tailored to this threat model. By enforcing protocol-level isolation to disentangle retrieval effects and categorizing model outputs into four failure modes (gold, hijacked, refusal, and semantic drift), the authors establish a quad-class assessment scheme along with clean-to-poisoned transition matrices. Experiments spanning five readers (7B–120B parameters), two retrievers, 3,145 questions, and 2,982 Sybil groups reveal that polymorphic attacks achieve a 22.8% hijacking rate, an 18.8-percentage-point increase over monomorphic attacks. Moreover, conventional Attack Success Rate (ASR) metrics critically overlook refusal and drift risks: models with identical ASR exhibit performance gaps exceeding 16 percentage points, exposing a significant blind spot in current RAG evaluations.