LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning

2026-08-02

AI Summary

Existing benchmarks struggle to evaluate the capability of multimodal large language models in complex reasoning that jointly involves multiple charts. To address this gap, this work proposes the first standardized evaluation framework specifically designed for multi-chart association and multi-step reasoning. Leveraging a latent-graph-guided synthesis approach, the authors construct a visual question answering benchmark comprising an average of 6.5 charts and 31.2 questions per instance, ensuring data consistency while enabling systematic assessment of model reasoning patterns, tool utilization, and robustness to image perturbations. Experiments across ten state-of-the-art models reveal a significant and volatile performance degradation as reasoning complexity increases, highlighting critical challenges in current multi-chart reasoning capabilities.

Abstract

Multimodal large language models (MLLMs) are rapidly evolving with expanded context windows and stronger reasoning capabilities, enabling multi-chart understanding and multi-step inference. These abilities are increasingly important as MLLMs are adopted in complex agentic tasks. However, existing benchmarks largely emphasize single-chart perception, while simple chart-to-chart connections are insufficient to evaluate these capabilities. To capture multi-chart complexity while ensuring consistency and validity, we design a synthesis pipeline supported by latent graphs. Building on this pipeline, we introduce LongChart, a benchmark whose VQA sets contain an average of 6.5 images and 31.2 questions. We evaluate 10 state-of-the-art MLLMs and examine three factors that influence performance: reasoning patterns, auxiliary tools, and robustness to image perturbations. Our results show that MLLM accuracy decreases and varies substantially as computational complexity increases, highlighting directions for future research in multi-chart reasoning.

Problem

• multi-chart reasoning
• multimodal large language models
• visual question answering
• complex reasoning
• benchmark evaluation

Innovation

• multi-chart reasoning
• latent graph
• synthesis pipeline
• multimodal large language models
• visual question answering