Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy

2026-07-16

AI Summary

This study addresses the lack of systematic evaluation of multimodal large language models (MLLMs) on scientific visualization comprehension, as existing benchmarks predominantly focus on conventional charts. To bridge this gap, the authors introduce scientific visualization literacy as a critical evaluation dimension and construct a benchmark dataset comprising 49 questions spanning 18 scientific image categories, 8 visualization techniques, and 11 task types. Using a closed-world protocol, they evaluate six prominent MLLMs and compare their performance against 485 human participants. Results reveal that Gemini surpasses the average human performance overall, while open-source models generally lag behind. Models exhibit relative strength in interpreting scientific illustrations and spatial reasoning tasks but show pronounced weaknesses in handling texture- or integration-based visualizations and quantitative estimation. This work establishes a fine-grained evaluation framework and empirical baseline for assessing the scientific reasoning capabilities of MLLMs.

Abstract

Multimodal large language models (MLLMs) are increasingly used to interpret visualizations, yet current evaluations remain largely chart-centric and provide limited evidence of understanding of scientific visualization (SciVis). We benchmark six MLLMs on the scientific visualization literacy assessment test, a standardized SciVis literacy assessment comprising 49 items based on 18 scientific visualizations and illustrations, spanning 8 techniques and 11 task types. We evaluate three closed-source and three open-source models under a closed-world protocol and compare their performance using data from 485 human participants. Results show that current MLLMs do not exhibit uniform SciVis literacy. Gemini is the strongest model overall, exceeding the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline. Performance is highly uneven across techniques and tasks: models perform best on scientific illustration, search, and spatial understanding, but struggle on texture-based and integration-based visualizations and on quantitative estimation. Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation. These findings position SciVis literacy as a necessary benchmark dimension for evaluating multimodal AI systems. Our code and model outputs are publicly available at https://github.com/patdmp/mllm-scivis-lit-benchmark.

Problem

• scientific visualization
• multimodal large language models
• visualization literacy
• benchmarking
• SciVis understanding

Innovation

• scientific visualization literacy
• multimodal large language models
• benchmarking
• visualization understanding
• human-AI comparison