VESTA: Visual Exploration with Statistical Tool Agents

2026-05-29

AI Summary

Existing automated scientific modeling approaches lack effective capabilities for model fitting and diagnostics in complex statistical tasks. This work proposes an intelligent agent framework that integrates vision and language models to actively explore and iteratively refine statistical models through hypothesis-driven visualization and the dynamic generation and reuse of high-order data transformations and statistical tests. The framework introduces a novel, dynamically expanding diagnostic toolkit mechanism, coupled with a context accumulation strategy, which substantially enhances modeling performance. Evaluated on the DAWN benchmark, the method outperforms current systems—particularly excelling in complex domains such as astronomy—and generates tools that span a broader range of diagnostic categories, thereby effectively supporting model inference.

Abstract

Fitting quantitative models to data is a central step in scientific workflows, yet it remains one of the least automated. Recent agent-based systems leverage language and vision-language models (VLMs) to iteratively propose and refine statistical models, but these systems struggle on more challenging modeling tasks. To address these limitations, we introduce VESTA: Visual Exploration with Statistical Tool Agents, a framework that equips VLMs with a dynamically growing exploration toolkit to guide model refinement through data transformations, hypothesis-driven visualizations, and robust statistical tests. Unlike prior systems that rely on iterative critique alone, VESTA actively explores data before and during refinement by selecting or creating diagnostic tools, which accumulate in the model's context and can be reused later. We evaluate VESTA against established baselines in three toolkit configurations: no tools, static expert-written tools, and dynamic model-written tools. To support this evaluation, we introduce DAWN (Dataset for Automated Workflows and Numerical Modeling), a benchmark targeting distribution fitting and time series modeling with varying difficulty tiers, and culminating in real-world astronomy tasks including modeling initial mass functions and gravitational-wave chirp signals. We find that VESTA's dynamic tool creation outperforms prior agentic pipelines, with the largest gains on complex and domain-specific tasks. We further show that dynamically generated tools are substantially more sophisticated than those produced by existing visual tool-creation systems, covering more diagnostic categories per function and strongly preferring visual outputs that the VLM critic can reason over directly.

Problem

• statistical modeling
• agent-based systems
• visual exploration
• data transformation
• model refinement

Innovation

• dynamic tool creation
• visual exploration
• statistical modeling
• vision-language models
• agent-based systems