DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

2026-07-24

AI Summary

Current evaluations of large language model (LLM) agents for database administration lack production-level fidelity and suffer from inconsistent tasks, platforms, and metrics, hindering meaningful cross-study comparisons. To address this, this work proposes DBA-Bench, a high-fidelity benchmark for database operations that introduces, for the first time, production-grade realism, outcome-oriented evaluation, and controllable scenario replay, covering four critical real-world operational gaps. Built on PostgreSQL instances under active workload, DBA-Bench integrates multi-source time-series and log observations, snapshot rollback, deep annotations, and an automated evaluation framework, encompassing 106 scenarios across seven task categories. In 848 automated runs, the best-performing agent achieved a safety pass rate of only 17.9%, starkly lower than human DBAs’ 93.4%, exposing significant limitations of current approaches in complex, safety-critical repair tasks.

Abstract

LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation and production operations: live-environment fidelity (multi-turn read-write interaction with a running database); observation-space scale and complexity (causal diagnosis across thousands of time series, business logs, and concurrent activity); solution-space openness (multiple remediations with different operational trade-offs); and scenario complexity and coverage (faults cascading across internal mechanisms and operational domains). We present DBA-Bench, a benchmark addressing these gaps through production fidelity, outcome-first evaluation, and controlled scenario reproducibility. It uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations; defines success by measurable recovery or fault elimination under safety constraints; and restores snapshots with scenario-specific checks before each run. The benchmark contains 106 scenarios across seven task domains, with two public difficulty labels based on reference-path diagnostic depth and environmental complexity. We evaluate nine baseline groups, including six foundation-model systems, two GPT-5.5-backed database agents, and a Human DBA reference. Across 848 automated runs, Diagnosis, Outcome, and Safe Pass rates are 32.7%, 19.6%, and 12.4%; the best automated baseline reaches 17.9% Safe Pass versus 93.4% for the Human DBA reference. Automated Safe Pass falls from 19.6% on Easy scenarios to 7.6% on Hard scenarios, underscoring the difficulty of safe end-to-end remediation.

Problem

• LLM-based database agents
• production-fidelity benchmark
• multi-turn read-write interaction
• causal diagnosis
• cascading faults

Innovation

• production-fidelity benchmark
• LLM-based database agents
• multi-turn read-write interaction
• causal diagnosis
• outcome-first evaluation