ChartAnno: Evaluating MLLMs for Chart Annotation Generation

2026-08-04

AI Summary

This work addresses the challenge of chart annotation generation, which requires integrated understanding of chart semantics, inference of communicative intent, and generation of appropriate textual or graphical elements. Despite the growing capabilities of multimodal large language models (MLLMs), their performance on this task has lacked systematic evaluation. To bridge this gap, we introduce ChartAnno, the first benchmark comprising 1,200 real-world charts paired with their source code and three-tiered instructions, supporting three input modalities: code-only, code-plus-image, and image-only. Through comprehensive automatic evaluation and ablation studies, we find that current MLLMs still face significant bottlenecks in abstract intent reasoning. While proprietary models generally outperform open-source counterparts, the performance gap is narrowing. Moreover, concrete instructions substantially improve annotation quality, and image inputs provide modest gains on design-oriented metrics.

Abstract

Multimodal large language models (MLLMs) have made significant progress in chart understanding, generation, and editing, but their ability to annotate existing charts remains underexplored. Annotating charts is a common yet challenging communicative task, requiring models to infer intended messages, interpret chart semantics, and place appropriate textual or graphical elements. To address this gap, we introduce ChartAnno, a benchmark for evaluating MLLMs on chart annotation generation. It contains 1,200 real-world charts with paired code and annotation instructions across three levels of instruction specificity. We evaluate 10 representative MLLMs under two primary input settings: (1) chart code alone and (2) both chart code and chart image, and further include a chart image-only ablation study. Results show that proprietary models remain stronger overall, although large-scale open-source models narrow the gap. More specific instructions improve annotation quality, while inferring abstract intent remains most difficult for current MLLMs. Providing chart images brings limited overall gains, with improvements mainly appearing in design-related metrics. These findings highlight chart annotation generation as a challenging task requiring semantic grounding and effective annotation design. Code and data will be released in a future version.

Problem

• chart annotation
• multimodal large language models
• semantic grounding
• annotation generation
• chart understanding

Innovation

• chart annotation
• multimodal large language models
• benchmark evaluation
• semantic grounding
• instruction specificity