JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

2026-09-28

AI Summary

This study addresses the limitation of existing embodied reasoning benchmarks, which overlook the need for actively acquiring dispersed evidence under restricted fields of view, thereby allowing models to generate plausible yet visually unsupported answers. To this end, we construct an active visual reasoning benchmark grounded in real-world robot data, introducing a structured question generation engine and an explicit observation-graph world model approach. This framework evaluates the capacity of agents to reason by requesting specific viewpoints and temporal observations while providing corresponding visual evidence. Experiments reveal a significant discrepancy between answer accuracy and evidence accuracy in current vision-language models (VLMs), underscoring the necessity of active evidence-aware evaluation for embodied visual reasoning.

Abstract

In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be distributed across time, viewpoint, and interacting objects. A model may therefore give a plausible answer without ever observing the relevant object, time, or view that supports it. Current visual reasoning benchmarks largely evaluate passive observations and final answers, overlooking settings that require active reasoning and evidence acquisition. We introduce JRDB-AVR, a benchmark derived from existing real-world JRDB robotics data through a structured question-generation engine that turns this gap into an explicit evaluation: an embodied agentic system receives a visual reasoning question, requests bounded observations by timestamp and viewing angle, and is evaluated on both the final answer and the grounded visual evidence supporting it. The benchmark contains diverse questions over multiple real-world environments involving temporal search, viewpoint selection, and human-oriented compositional reasoning. We also introduce JRDB-AVR-Agent, a reference active reasoning agentic method that maintains an explicit observation-grounded graph-based world model and answers through solving. Experiments reveal a substantial gap between answer accuracy and evidence accuracy in current baselines, showing that current VLMs can produce unsupported correct answers and that active evidence-aware evaluation is necessary for embodied visual reasoning. Code and benchmark are available at https://github.com/ControlNet/JRDB-AVR.

Problem

• embodied visual reasoning
• active reasoning
• evidence acquisition
• benchmark evaluation
• visual language models

Innovation

• Active Visual Reasoning
• Embodied Agents
• Evidence-grounded Evaluation
• Graph-based World Model
• Benchmark