AI Security Leaderboard: Methodology, Results and Minimal Standard

2026-08-03

AI Summary

This work addresses the absence of open, systematic benchmarks for evaluating the safety defenses of AI models against malicious misuse, such as CBRNE and cyber attacks. It introduces FAR.AI Safety Defense Minimum Standard v1.0, establishing a taxonomy of 67 static jailbreaking techniques to construct a large-scale combinatorial attack space. A three-stage red-teaming funnel identifies universal jailbreak prompts, while a cost-metric model—incorporating right-truncated lower-bound estimation—quantifies attack overhead. For the first time, this study establishes a reproducible, public AI safety benchmark, introduces the concept of “universal jailbreaks,” and proposes the “cost-to-jailbreak” metric. Results reveal stark disparities among leading models: Grok 4.5 and Gemini 3.1 Pro exhibit 63 to 385 universal jailbreaks, respectively, whereas Claude Fable 5 and GPT-5.6 Sol show none, with attack costs differing by over two orders of magnitude.

Abstract

Frontier AI model developers increasingly rely on layered safeguards to prevent catastrophic misuse, but little public evidence exists on how much protection these safeguards provide, or how consistently across developers. We introduce the FAR.AI Minimal Standard for Safeguards, Version 1.0: a taxonomy of 67 readily accessible static jailbreak techniques, a method for composing them into a very large attack space, and a benchmark of flagship models against a sample of it. We evaluate Claude Fable 5, GPT-5.6 Sol, Gemini 3.1 Pro, and Grok 4.5 on two complementary datasets totalling 360 attacker goals spanning chemical, biological, radiological/nuclear and explosive (CBRNE) threats and offensive cyber, using a three-stage funnel to identify universal jailbreaks: single prompt templates that elicit operationally compliant responses on over 75% of a domain's goals. We also introduce a cost-to-jailbreak metric that models attacker spend directly, with right-censored lower bounds where no universal jailbreak was found. Robustness is highly uneven: the cost to break these models varies over a hundredfold. Random search over our technique pool found 63 universal jailbreaks against Grok 4.5 and 18 against Gemini 3.1 Pro, at an average cost of roughly $58 and $278 per jailbreak found; expert-guided composition raised these to 385 and 231. Neither Claude Fable 5 nor GPT-5.6 Sol yielded any universal jailbreak under either strategy. Because meeting the Minimal Standard requires only defenses already publicly described and deployed in production elsewhere, these gaps appear closable with current techniques. We recommend defense-in-depth combining reasoning, activation, and input/output monitoring. Results are maintained at leaderboard.far.ai.

Problem

• AI security
• safeguards
• jailbreak
• robustness
• Minimal Standard

Innovation

• AI safety benchmarking
• jailbreak evaluation
• universal jailbreak
• cost-to-jailbreak metric
• layered safeguards