Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks

2025-10-07

AI Summary

The absence of standardized benchmarks hinders systematic evaluation of AI models’ understanding of scatter plots. Method: We introduce ScatterBench, the first comprehensive benchmark for scatter plot comprehension, comprising over 18,000 synthetic samples generated via six data-generation patterns and seventeen chart designs. We propose an N-shot prompting-driven multi-task evaluation framework covering four spatial reasoning tasks: cluster counting, outlier detection, bounding box localization, and centroid coordinate prediction. Results: Experiments reveal that state-of-the-art closed-source models (e.g., GPT-4o, Gemini 2.5 Flash) achieve >90% accuracy on cluster counting but underperform significantly on localization tasks (F1 < 50%). Aspect ratio and color encoding are identified as critical visual factors substantially affecting model performance. This work provides the first systematic characterization of current AI models’ capabilities and design sensitivities in scatter plot spatial reasoning.

Technology Category

Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal ReasoningMachine Learning: Large Multimodal Models (LMMs)Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

AI models are increasingly used for data analysis and visualization, yet benchmarks rarely address scatterplot-specific tasks, limiting insight into performance. To address this gap for one of the most common chart types, we introduce a synthetic, annotated dataset of over 18,000 scatterplots from six data generators and 17 chart designs, and a benchmark based on it. We evaluate proprietary models from OpenAI and Google using N-shot prompting on five distinct tasks derived from annotations of cluster bounding boxes, their center coordinates, and outlier coordinates. OpenAI models and Gemini 2.5 Flash, especially when prompted with examples, are viable options for counting clusters and, in Flash's case, outliers (90%+ Accuracy). However, the results for localization-related tasks are unsatisfactory: Precision and Recall are near or below 50%, except for Flash in outlier identification (65.01%). Furthermore, the impact of chart design on performance appears to be a secondary factor, but it is advisable to avoid scatterplots with wide aspect ratios (16:9 and 21:9) or those colored randomly. Supplementary materials are available at https://github.com/feedzai/biy-paper.

Problem

• Addressing the lack of benchmarks for scatterplot-specific AI model tasks
• Evaluating AI models on scatterplot analysis tasks using synthetic datasets
• Assessing model performance on cluster counting and outlier localization tasks

Innovation

• Created synthetic dataset with 18000 annotated scatterplots
• Evaluated AI models using N-shot prompting on five tasks
• Benchmarked cluster counting and outlier detection performance