DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

2026-08-04

AI Summary

Existing benchmarks struggle to uniformly evaluate data agents’ capability for verifiable analysis in heterogeneous workspaces, particularly lacking support for multimodal evidence integration, complete tabular outputs, and deterministic evaluation. This work proposes DataSpace—the first verifiable analysis benchmark enabling cross-lingual fusion of structured and unstructured multimodal data—and introduces the execution-driven DataSpace-Builder framework alongside a deterministic evaluation protocol. Key technical innovations include cross-lingual transformation, constraint-aware relational sampling, modality routing, artifact rendering, and header-agnostic column alignment. Experiments across six state-of-the-art multimodal models and five agent frameworks reveal a peak accuracy of 66.34%, with framework selection yielding performance gaps up to 15.36 points, demonstrating the benchmark’s challenge and unsaturated nature. DataSpace will serve as the official evaluation platform for KDD Cup 2026.

Abstract

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.

Problem

• heterogeneous workspaces
• data agents
• verifiable analytics
• tabular outputs
• benchmarking

Innovation

• Data Agents
• Heterogeneous Workspaces
• Verifiable Analytics
• Deterministic Evaluation
• Multimodal Benchmark