AnnoBench: A Benchmark for Visualization Annotation Generation

2026-07-28

AI Summary

Automated generation of visual annotations faces significant challenges in simultaneously satisfying visual, semantic, and stylistic constraints, compounded by the absence of a unified evaluation benchmark. This work proposes AnnoBench, the first structured benchmark for this task, built upon professional data journalism and visualization repositories. It systematically integrates multi-format input representations, controllable semantic contexts, and prompts at varying levels of granularity. The study introduces an innovative visual-language model (VLM)-as-a-judge paradigm to enable scalable and reproducible automatic evaluation that closely aligns with human judgments. Through four ablation studies, the authors uncover the critical influence of input representation, contextual information, prompt specificity, and model selection on annotation quality, thereby establishing a foundational framework for future research in automated visual annotation.

Abstract

Annotation is among the most demanding visualization tasks to automate, as it simultaneously requires correctly navigating visual, semantic, and stylistic constraints. Failure to meet any of these conditions severely undermines the utility of an annotation, rendering it challenging to read, inaccurate, or visually discordant. Despite a growing body of annotation tools and automations, no existing benchmark or evaluation framework tests whether these conditions are met because of their scope and annotation not being the focus of their studies. We introduce AnnoBench, a benchmark for visualization annotation that materializes the inherent challenges of this domain in a structured and testable manner. AnnoBench pairs visualizations from professional data journalism and visualization galleries with annotation tasks, spanning four representation formats, five chart description conditions, and two prompt specification levels. The benchmark is executed via VLM-as-a-judge, using models aligned with manual human assessment. We evaluate the benchmark via four one-factor-at-a-time experiments, exploring the effects of input representation, semantic context, and prompt specificity, and model selection on annotation quality. This work provides a foundation for advancing annotation automation, tooling, and visualization-generation pipelines.

Problem

• visualization annotation
• benchmark
• annotation automation
• evaluation framework
• visual semantics

Innovation

• annotation benchmark
• visualization annotation
• VLM-as-a-judge
• multimodal evaluation
• automated annotation