Fail Loudly: An Auditable Runtime for Agentic Data Analysis

2026-09-26

AI Summary

This study addresses the problem of silent errors caused by erroneous choices in agentic data analysis by proposing RADAR, a runtime framework that introduces a novel "fail-loud" mechanism. This approach delegates semantic interpretation to large language models while restricting the runtime to structural validation and feedback. Through evidence-preserving exploration, typed operator logging, and conflict detection algorithms, the framework ensures transparent and auditable analytical decisions, effectively preventing error propagation. Evaluations on benchmarks including KramaBench demonstrate that RADAR outperforms the strongest baselines by 35.9%, 14.0%, and 59.3% across respective metrics, significantly enhancing the reliability of agentic data analysis.

Abstract

Large language models (LLMs) have enabled data-science agents to automate multi-step analyses over heterogeneous files. However, incorrect choices regarding data sources, scope, or statistical definitions often lead to silent errors: computations execute successfully but produce plausible yet incorrect outputs that fail to answer the intended question. To mitigate this, we present RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback. RADAR operates through three core mechanisms. First, an evidence-preserving exploration module retrieves task-relevant content while retaining source locations and observation coverage. Next, the runtime uses typed operators to record the agent's declared inputs, operation arguments, and resulting observations. Finally, runtime validation checks proposed operations against these observations. When a conflict is detected, the runtime rejects the operation or provides diagnostic feedback, allowing the agent to revise its choices before errors propagate. This design enables agents to fail loudly while leaving semantic interpretation to the LLM. On KramaBench, RADAR achieves overall scores of 0.723 with full source retrieval and 0.747 with gold sources supplied, corresponding to relative gains of 35.9% and 28.8% over the strongest baselines. Beyond KramaBench, RADAR achieves relative performance gains of 14.0% on DA-Code and 59.3% on DABStep, demonstrating its applicability across diverse agentic data-analysis workflows.

Problem

• Large Language Models
• Data-Science Agents
• Silent Errors
• Agentic Data Analysis
• Runtime Auditing

Innovation

• Auditable Runtime
• Agentic Data Analysis
• Fail Loudly
• Runtime Validation
• Evidence-preserving Exploration