RAPID: Robot Agentic Programming from Demonstrations

2026-09-24

AI Summary

This study addresses the challenge of automatically generating generalizable robotic manipulation programs from a single visual demonstration. To this end, it proposes an agent-in-the-loop automated framework that infers task specifications and action primitives from demonstrations to enable code generation and iterative refinement. The core innovation lies in adopting an object-centric relational representation that emphasizes policy structure over low-level motion trajectories, integrating relational constraint reasoning with interactive environmental verification for trajectory optimization. The proposed method successfully executes complex manipulation tasks on both simulated and real-world robotic arms. It demonstrates strong generalization across variations in object poses, shapes, and scene configurations, substantially enhancing the cross-scenario reusability of the generated programs.

Abstract

Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.

Problem

• robot programming
• learning from demonstration
• coding agents
• manipulation tasks
• program generalization

Innovation

• Coding Agents
• Programming from Demonstrations
• Object-centric Representation
• Trajectory Optimization
• Nonprehensile Manipulation