CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation

2026-09-27

AI Summary

This study evaluates the capacity of general-purpose multimodal large models to translate visual understanding into embodied manipulation. We propose a "code-as-policy" agent framework that requires no fine-tuning, dedicated perception modules, or predefined policies. By leveraging only a shared robot API and RGB visual feedback, the framework prompts large models to generate and execute manipulation code, enabling end-to-end decision-making with closed-loop iterative correction. Experimental results demonstrate that the optimal configuration successfully completes 22 out of 25 tasks, achieving an average success rate of 73.3%. Furthermore, this work systematically identifies critical failure modes, such as spatial misalignment. Overall, these findings provide a valuable benchmark and empirical evidence for advancing general-model-driven embodied intelligence.

Abstract

How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation tasks that evaluates this capability through agentic Code-as-Policy. Without task-specific fine-tuning, demonstrations, external specialist perception or grasp modules, privileged scene state, or predefined task policies, agents should select visual evidence, form task-relevant 3D estimates, construct manipulation targets, and iteratively execute and revise their policies. A shared robot API provides RGB observations, calibrated geometric operations, robot feedback, and bounded motion, leaving task-dependent decisions to the evaluated agent. Fixed task instances, resource budgets, and a hidden physical-outcome verifier support controlled comparisons across models and harness configurations. Extensive evaluations across nine configurations and 675 attempts achieve success rates ranging from 2.7% to 73.3%. The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts, demonstrating the best performance while still leaving substantial room for improvement. Trajectory analyses reveal difficulties in spatial alignment, object retention, and completion judgment, including task failures despite successfully completed motions. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining typical failure scenarios in that process.

Problem

• Embodied Manipulation
• Multimodal Models
• Code-as-Policy
• Benchmark Evaluation
• Agentic Reasoning

Innovation

• Code-as-Policy
• Embodied Manipulation
• Benchmark
• Multimodal Models
• Agentic Framework