Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

2026-07-29

AI Summary

Current safety benchmarks for AI agents lack validity validation, often conflating model capabilities with alignment. This study conducts a systematic audit of four prominent benchmarks—R-Judge, InjecAgent, AgentHarm, and AgentDojo—evaluating 22 models under a unified protocol and constructing a composite capability metric using MMLU and GPQA to control for competence confounds. Through correlation analysis, leave-one-out cross-validation, and cluster-based bootstrapping, the work reveals for the first time that safety scores are significantly influenced by metric design choices (e.g., F1’s bias toward always-positive strategies) and small-sample panel effects. Findings show that capability positively correlates with task success (ρ = +0.60) but negatively with misalignment-related safety failures (ρ = −0.44). All benchmarks primarily measure harmful compliance and exhibit only aggregative validity, with AgentHarm showing the strongest post-capability-adjusted association with jailbreaking safety (ρ = +0.72). The paper argues that rigorous safety evaluation must explicitly specify benchmarks, metrics, target behaviors, and model sets.

Abstract

Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent's safety. We treat four of them (R-Judge, InjecAgent, AgentHarm, AgentDojo) as measurements to be validated, running each under its official implementation and author-provided scorer on up to 22 models, with MMLU and GPQA measured by us under one protocol as a capability composite. The metric is the first problem. On any binary trace-judgment benchmark scored by $F_1$, an ``always positive'' policy attains $F_1 = 2π/(1+π)$; on R-Judge that is $0.690$, above five of the 21 models that actually discriminate. The three broad-coverage benchmarks then rank the same 18 models differently, and the trade-off behind that disagreement is a small-panel artifact: R-Judge specificity against AgentHarm safety correlates $-0.64$ at $n{=}7$ and $+0.02$ at $n{=}18$, and a quarter of random size-7 subsets reach $|ρ| \geq 0.5$ around that near-zero value. Held-out validity turns on which outcome you pick. Capability predicts task success ($ρ{=}{+}0.60$) but correlates negatively with misalignment safety ($ρ{=}{-}0.44$, $n{=}21$). On their paired $n{=}20$ panel, the corresponding contrast is $Δ{=}{-}1.00$ (95% CI $[-1.48, -0.49]$, $p<0.001$), and it survives leave-one-organization-out and organization-clustered bootstrap analyses. On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant. \mbox{AgentHarm} shows the strongest held-out association, $ρ{=}{+}0.72$ with three-template jailbreak safety after controlling capability. But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety. Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.

Problem

• agent safety
• benchmark validity
• safety evaluation
• capability-safety trade-off
• misalignment

Innovation

• agent safety benchmarks
• validity audit
• capability-misalignment trade-off
• convergent validity
• evaluation metrics