Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation

2026-09-25

AI Summary

This study addresses the challenge of geometric and task-level incompatibilities among independently generated code components from vision-language models (VLMs), which hinders the construction of unified robotic manipulation systems. To this end, we propose RIVET, a framework that introduces a novel object-centric shared representation integrating 6D poses with relational graphs. This representation guides VLMs to collaboratively generate executable code modules for perception and planning while supporting offline reuse without repeated generation. Experimental evaluations on stacking and reassembly tasks in both simulated and real-world robotic settings demonstrate that the proposed framework achieves an overall success rate of 83% by leveraging pre-generated offline systems. These results effectively validate the feasibility and efficiency of employing a shared representation to drive the collaborative generation of multi-module code for robotic manipulation.

Abstract

Building a robotic manipulation system requires connecting perception, planning, and control through carefully designed representations and interfaces. VLM code generation offers a way to automate this construction, but independently generated components may operate on incompatible geometric and task-level information. We present Representation-guided Integration of VLM-generated Executable Task programs (RIVET), a framework for generating complete manipulation systems around a shared object-centric representation. The representation combines per-object 6D poses, which preserve the metric information required for action grounding, with a relation graph that exposes the task-level structure required for planning. Guided by this representation, a VLM generates cooperating perception, rendering, relation-inference, and planning programs, each combining task-specific computation with available packages where useful. The resulting programs are authored once for a manipulation domain and reused on unseen start and goal configurations without code regeneration. We evaluate RIVET on cube stacking, tangram rearrangement, and three-dimensional assembly in simulation and on a physical robot, where we achieve 83% overall success rate in the real world by reusing offline-generated systems. Our results demonstrate that representation-guided program generation can adapt a common manipulation framework to tasks with different geometric, relational, and sequential requirements.

Problem

• Robotic Manipulation
• VLM Code Generation
• System Integration
• Representation Compatibility

Innovation

• Representation-guided generation
• Vision-Language Model (VLM)
• Object-centric representation
• Executable program integration
• Robot manipulation