VLMs Can Describe, But Not Measure: Object-Centric Scene Understanding for Robotic Manipulation

2026-09-23

AI Summary

本文针对机器人在未知环境中的操作问题，提出了一种基于视觉-语言模型的模块化感知框架，通过结合RGB-D数据和深度信息来提高场景理解和定位准确性。

Abstract

Robotic operation in previously unseen environments requires both semantic understanding and reliable metric information. While vision--language models (VLMs) provide strong semantic capabilities, their geometric estimates remain less reliable. In this paper, we propose a VLM-driven, modular perception framework for scene understanding using off-the-shelf approaches. Starting from a single RGB-D observation, the scene is segmented into object-level regions, annotated by a VLM, and grounded with depth information to construct a task-independent object-centric representation. Experiments on 151 tabletop scenes show that the proposed decomposition preserves strong semantic performance while substantially improving localization and depth estimation over direct VLM inference. The resulting representation is also integrated with a task-planning framework for robotic execution.

Problem

• Robotic Operation
• Semantic Understanding
• Metric Information
• Vision-Language Models
• Geometric Estimates

Innovation

• VLM-driven
• modular perception framework
• object-centric representation
• semantic understanding
• depth estimation