CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization

2026-06-02

AI Summary

This work addresses the challenge of learning continuous, semantically meaningful latent action representations from videos without action labels and constructing world models capable of supporting planning and imitation learning. To this end, the authors propose an end-to-end self-supervised framework that, for the first time, jointly learns structured latent actions and a dynamics-aware world model without requiring any action annotations. The approach integrates adversarial latent regularization with diffusion-based video generation to simultaneously optimize the latent action representation and environmental dynamics. Experiments demonstrate that the learned latent actions exhibit strong semantic coherence and cross-task transferability, significantly outperforming existing methods in goal-directed planning and imitation learning tasks.

Abstract

We introduce CLAW, a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free videos. Our approach leverages adversarial latent regularization and diffusion-based video generation to capture structured and semantically meaningful action representations while modeling rich, predictive environment dynamics, without relying on any action labels or annotations. By simultaneously training the Latent Action Model and world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone. We show that the resulting latent action world model supports both imitation learning from observation and goal-directed planning. In imitation learning, latent actions extracted from raw videos enable behavior cloning. For planning, CLAW generates sequences of latent actions and maps them to executable actions to reach desired goals. Extensive experiments across diverse tasks and embodiments demonstrate that CLAW produces semantically meaningful latent action representations, supports effective action transfer, and enables planning and imitation from observation, outperforming existing methods.

Problem

• latent actions
• world models
• action-free videos
• imitation learning
• goal-directed planning

Innovation

• adversarial latent regularization
• continuous latent actions
• action-free video learning
• diffusion-based world model
• self-supervised imitation