PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration

2026-07-17

AI Summary

Existing action-conditioned world models struggle to simultaneously achieve physical plausibility, action consistency, and visual fidelity (PAV), and exhibit limited generalization under out-of-distribution actions. This work proposes a novel approach based on a pretrained latent-space world model that, for the first time, explicitly integrates the PAV triad into a reward-driven reinforcement learning framework. It introduces an unsupervised noise-action exploration mechanism that operates without paired video supervision, jointly optimizing in-distribution trajectory modeling and out-of-distribution generalization. Experiments demonstrate that the method improves average performance by 5.6% across multiple benchmarks, significantly enhancing generation quality and model robustness. Furthermore, when employed as a policy evaluator, it effectively mitigates value overestimation bias.

Abstract

Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV-while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World. Code: https://github.com/Social-AI-Studio/PAVXploreRL

Problem

• world model
• action-conditioned
• out-of-distribution generalization
• physical plausibility
• visual fidelity

Innovation

• world model
• reinforcement learning
• action exploration
• physical plausibility
• out-of-distribution generalization