ORCA-bench: How Ready Are Language Model Agents for Oncall?

2026-07-30

AI Summary

This study evaluates the capability of general-purpose large code models to perform root cause analysis (RCA) in real-world production environments—a task requiring integration of ambiguous user reports with multi-source, heterogeneous operational data, including metrics, logs, traces, and source code. To this end, we introduce ORCA-bench, a high-fidelity oncall benchmark built upon an OpenTelemetry-instrumented microservice system and comprising 1,079 structured RCA tasks. We present the first systematic evaluation of large models on a publicly available, reproducible dataset of 50GB of six-day production-grade telemetry. Using expert-annotated ground-truth symptoms and a high-agreement LLM-as-judge mechanism, we find that even the best current models achieve only 25.3% and 10.0% accuracy on medium- and high-difficulty tasks, respectively. Performance degrades significantly—and hallucinations increase—when source code access is removed or weaker models are used, revealing a substantial gap between existing models and production-grade reliability.

Abstract

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.

Problem

• oncall
• root cause analysis
• language model agents
• production reliability
• microservice debugging

Innovation

• ORCA-bench
• root cause analysis
• language model agents
• production telemetry
• oncall automation