Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems

2026-06-06

AI Summary

Current evaluations of visual language models (VLMs) for symbolic reasoning are largely confined to English-centric mathematical tasks, lacking multilingual coverage and visual grounding, thereby limiting their ability to comprehensively assess robustness in STEM domains. This work introduces the first dynamic, multilingual, visually grounded symbolic STEM benchmark, spanning five disciplines and seven languages. Leveraging executable templates, it generates semantically equivalent yet syntactically diverse problem instances, each accompanied by reference solutions and step-by-step reasoning traces. We propose worst-case accuracy and step-level F1 metrics to evaluate 17 state-of-the-art VLMs, revealing a substantial performance gap between average and worst-case scenarios. Larger models demonstrate greater multilingual robustness, while attention analyses uncover cross-lingual imbalances in the alignment between visual and textual tokens.

Abstract

Symbolic benchmarks have emerged as a key approach to assess model robustness under minor modifications to STEM-related questions. However, existing symbolic benchmarks mostly remain limited to mathematical reasoning, lack visual grounding, and are predominantly in English. In this work, we introduce Sci-Rho (Science Rhobustness), a dynamic benchmark for visually-grounded STEM problems spanning five subjects and seven languages, comprising 4,242 problem templates (606 per language) crafted by domain experts, including Olympiad medalists. Each template is implemented as executable Python code that generates diverse but equivalent problem instances by varying numerical values, visual patterns, geometric shapes, color schemes, and function types, resulting in 42,420 instances in total, each paired with reasoning steps and ground-truth solutions. We evaluated 17 state-of-the-art VLMs and discovered a noticeable gap between worst-case accuracy (defined as the proportion of problem templates that a model answers correctly across every generated variation) and average accuracy. We also discovered that smaller models show noticeable performance degradation across languages, whereas proprietary and larger models remain robust. Step-level evaluation reflects this same trend, revealing a significant gap between average F1 and worst-case F1 scores. Finally, our inspection of attention heads of a VLM reveals substantial cross-lingual variation in the relative attention allocated to image tokens compared to text tokens. Our work highlights the importance of evaluation beyond static benchmarks as a metric to measure the quality of VLMs.

Problem

• symbolic benchmarks
• visually-grounded reasoning
• multilingual STEM problems
• model robustness
• visual language models

Innovation

• visually-grounded
• symbolic benchmark
• multilingual
• dynamic evaluation
• visual language models