Chart-Supported or Model-Supplied? Examining MLLM-Generated Claims for Accessible Visualization

2026-07-27

AI Summary

This study addresses the tendency of multimodal large language models (MLLMs) to conflate chart-based evidence with speculative reasoning in generated visual explanations, thereby undermining their reliability. It presents the first systematic framework distinguishing between direct evidence (DIRECT), derived content (DERIVED), and speculative statements (SPECULATIVE) in MLLM outputs. Analyzing 1,224 descriptions produced by four leading MLLMs across 102 charts—using both automated numerical consistency checks and human annotations—the work introduces an evaluation framework tailored for accessibility-oriented visual descriptions. Findings reveal that providing accessibility-aware context significantly increases the proportion of direct claims and improves numerical accuracy, whereas relying solely on visual input yields no consistent benefit. Moreover, interpretations of real-world implications remain predominantly speculative.

Abstract

Multimodal large language models (MLLMs) can connect visualization patterns to external causes, consequences, and domain knowledge, but the evidential basis of these interpretations is often unclear. We present an exploratory study of 102 visualizations from four sources, three MLLMs, and four input conditions that vary access to the image, source-specific accessible chart context, and withheld-context framing. Across 1,224 descriptions, we analyze model-attributed DIRECT, DERIVED, and SPECULATIVE labels and conduct an automated audit of numeric agreement. Accessible chart context shifted Gemini and GPT toward DIRECT claims and improved numeric agreement for some models. Adding the image to the full context did not yield a consistent numeric benefit, and the withheld-context prompt did not reliably increase cautious language. The prompt-defined Real-World Significance section remained predominantly SPECULATIVE. These results motivate accessible description systems that distinguish claims supported by supplied evidence from model-supplied interpretation

Problem

• multimodal large language models
• accessible visualization
• evidential basis
• model interpretation
• claim classification

Innovation

• Multimodal Large Language Models
• Accessible Visualization
• Evidence-Based Claim Classification
• Numeric Agreement Audit
• Model Interpretation Transparency