SceneActBench: Can Agents Act on the 3D Scenes They See?

2026-07-24

AI Summary

Existing 3D benchmarks are largely confined to textual responses or single-object manipulation, making them inadequate for evaluating agents’ visually guided action capabilities in complex multi-object scenarios. To address this limitation, this work proposes SceneActBench—the first comprehensive benchmark specifically designed to assess agent action proficiency within complete 3D scenes. It encompasses five task categories, 210 source scenes, and 520 task instances. The benchmark employs a unified agent–environment interaction loop, taking as input images, video frames, and optionally 3D assets, and evaluates output fidelity against ground truth using task-specific geometric metrics. Experiments across 11 vision-language models reveal overall scores ranging from 38.6 to 50.2, with no model demonstrating consistent performance across all tasks, thereby exposing critical bottlenecks in 3D embodied intelligence.

Abstract

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.

Problem

• 3D scenes
• agent action
• vision-language models
• benchmarking
• multi-object interaction

Innovation

• SceneActBench
• vision-language models
• 3D scene interaction
• agent-environment loop
• geometric evaluation metrics