RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

2026-09-28

AI Summary

This study addresses the limitation of existing Vision-Language-Action (VLA) benchmarks that predominantly assume explicit instructions, overlooking robots' capacity to infer implicit references from context. To this end, we propose RoboIRG-Bench, a benchmark designed to systematically evaluate the implicit reference grounding capabilities of VLA models. Methodologically, we formally define this task and construct an evaluation suite encompassing four distinct challenge categories. Building upon the RoboMME framework, we investigate the effectiveness of various memory mechanisms and external VLM-assisted architectures. Experimental results demonstrate that reasoning and spatial referencing constitute core bottlenecks for current models, while real-world robotic arm deployments further reveal significant performance degradation. This work fills a critical gap in evaluating implicit understanding within VLA systems, exposes substantial robustness deficiencies, and establishes new directions for embodied intelligence research.

Abstract

Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct, reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context. Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3 robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate language, perception, reasoning, and action.

Problem

• Implicit Referential Grounding
• Vision-Language-Action Models
• Robotic Manipulation
• Benchmark
• Referential Robustness

Innovation

• Implicit Referential Grounding
• Vision-Language-Action Models
• RoboIRGBench
• Referential Robustness Gap
• Robotic Manipulation