RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models

2026-06-01

AI Summary

Current vision-language-action (VLA) models struggle to accurately map high-level semantic instructions to physical actions when executing complex tasks, often relying on superficial shortcuts rather than genuine understanding. To address this limitation, this work introduces RoboSemanticBench, the first embodied evaluation benchmark specifically designed to assess the semantic grounding capabilities of VLA models. The benchmark employs multiple-choice question-driven grasping tasks—featuring both four-option and ten-option configurations—to isolate and measure semantic decision-making performance while controlling for grasping success rates. Experimental results reveal that state-of-the-art VLA models perform at or below random chance in selecting semantically correct targets, highlighting a significant gap between semantic comprehension and action prediction.

Abstract

Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB), an embodied benchmark for diagnosing semantic grounding in action prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction.

Problem

• semantic grounding
• action prediction
• vision-language-action models
• embodied reasoning
• robotic manipulation

Innovation

• semantic grounding
• vision-language-action models
• embodied benchmark
• action prediction
• RoboSemanticBench