World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

2026-07-29

AI Summary

This work addresses the limited generalization of imitation learning policies in novel scenarios and tasks by proposing a robotic planning system that integrates high-level reasoning from vision-language models (VLMs) with the physical awareness of a multi-task pose- and image-conditioned world model. The approach uniquely combines the semantic reasoning capabilities of VLMs with the physics-based simulation of action-conditioned world models to generate executable action sequences through imagined rollouts and iterative refinement. Evaluated under challenging conditions—including compositional tasks, novel environment layouts, and zero-shot transfer settings—the method significantly outperforms existing end-to-end approaches such as Vision-Language-Action (VLA) models and World Model Agents (WAMs), demonstrating superior generalization and enhanced capability in compositional task planning.

Abstract

Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io

Problem

• generalizable agents
• zero-shot generalization
• compositional tasks
• novel scenes
• task generalization

Innovation

• World Action Planner
• Vision-Language Models
• Action-Conditioned World Models
• Zero-Shot Generalization
• Iterative Planning