ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats

2026-05-31

AI Summary

Existing chart parsing models lack a unified evaluation benchmark that comprehensively covers multilingual inputs, diverse scenarios (digitally rendered, printed, and hand-drawn), and varied chart types—particularly diagrammatic structures like flowcharts—while incompatible output formats further hinder fair assessment. To address these gaps, this work introduces ChartBench, the first bilingual, comprehensive benchmark encompassing both quantitative and diagrammatic charts. Data quality is ensured through a human-in-the-loop, multi-stage annotation pipeline, and a format-agnostic semantic evaluation protocol based on triplets and directed graphs enables equitable model comparison. Evaluation of 26 state-of-the-art multimodal large language models reveals that closed-source models currently lead overall, though open-source systems are rapidly closing the gap; document parsing models struggle notably with diagrammatic structures; and radar charts and hand-drawn inputs remain persistent challenges, offering clear directions for future research.

Abstract

Charts are a primary medium for conveying quantitative and relational information, yet systematically evaluating chart parsing models remains difficult. Existing benchmarks focus on narrow chart types and leave diagrammatic structures such as flowcharts and mind maps largely unaddressed, while models produce outputs in incompatible formats, and datasets rarely include the printed or hand-drawn images encountered in practice. To address these issues, we introduce ChartArena, a comprehensive bilingual benchmark covering eight chart families spanning both numeric charts and diagrammatic structures, each evaluated across three visual scenarios: digital renderings, printed photos, and hand-drawn photos. The dataset is built via a human-agent collaborative annotation pipeline with multi-stage human verification to ensure annotation reliability. To enable fair cross-model comparison, we further design a format-agnostic evaluation protocol that maps heterogeneous outputs into two canonical semantic spaces, a normalized triple view and a directed graph view, and scores them with structure-aware metrics. Through extensive evaluation of 26 leading MLLMs, we observe three consistent findings: (i) frontier proprietary models such as Gemini 3.1 Pro lead overall, yet the strongest open-source systems are rapidly closing the gap; (ii) document parsing models handle numeric charts reasonably but fall sharply behind on diagrammatic structures; and (iii) expert chart parsers remain limited to narrow chart families. Across all models, radar charts and hand-drawn scenarios stay especially challenging. These findings show that ChartArena exposes clear capability gaps and provides a unified foundation for future progress. ChartArena is publicly available at https://github.com/pspdada/ChartArena.

Problem

• chart parsing
• benchmarking
• diagrammatic structures
• multimodal evaluation
• format heterogeneity

Innovation

• chart parsing
• format-agnostic evaluation
• semantic normalization
• human-agent collaborative annotation
• multilingual benchmark