Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

2026-07-29

AI Summary

This work addresses the absence of a public, unified, and queryable enterprise security environment for end-to-end evaluation of the trustworthiness and effectiveness of autonomous cyber defense agents. It presents the first open benchmarking framework tailored for autonomous enterprise defense, leveraging frozen snapshots of enterprise security states combined with dual-modality interfaces—text-to-SQL and native APIs—to enable reproducible and auditable assessments of agents’ investigative capabilities across multiple dimensions, including identity, cloud, and data security. The framework integrates relational snapshots, multi-vendor APIs, synthetic organizational datasets, and a multidimensional scoring mechanism. It has been successfully instantiated with two identity security packages and synthetic environments at varying scales, demonstrating its efficacy in the evaluation phase and laying the groundwork for future extension into remediation.

Abstract

Enterprises are moving toward autonomous cyber defense: agentic AI that builds situational awareness of an organization's security state and reasons from it to assessments, decisions, and actions. This rests on a holistic view of the enterprise's security state, the continuous, cross-vendor picture of identities, cloud and infrastructure, data, applications, and their configurations that security posture management assembles. As agents take on this work, what matters is not whether an agent can produce an answer but whether it should be trusted to. The field cannot yet answer this question. Real enterprise environments are private, cross-vendor, and deeply correlated, and none is exposed publicly as a shared, queryable target for evaluating such agents end to end. We call this the environment data gap. We present Open Security Benchmark (OSB), a framework that benchmarks agentic AI on this work. OSB surfaces a curated enterprise environment - a frozen, holistic view of the security state - and evaluates posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor's native API over a served instance of the same environment. Freezing the environment pins the target state as an immutable snapshot and anchors answers to a closed-form ground truth. OSB is built from five components: a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path that serves public comparison and private tenant evaluation from one substrate. We instantiate the framework with two identity-security packs and a family of synthetic-organization environment datasets spanning multiple scales, and chart its extension to further posture subdomains, investigation modalities, and defense stages from assessment toward remediation.

Problem

• autonomous cyber defense
• agentic AI
• security posture management
• enterprise security benchmarking
• environment data gap

Innovation

• Agentic AI
• Security Posture Management
• Open Security Benchmark
• Enterprise Cyber Defense
• Text-to-SQL