What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction

2026-06-05

AI Summary

This work investigates which pretraining signals induce action-relevant structural information in the latent spaces of video world models. Employing a unified inverse dynamics probing framework, the study systematically evaluates diverse pretraining strategies—including image-based self-supervision, video temporal modeling, autoencoders, diffusion models, and explicit dynamics models. The findings demonstrate that pretraining leveraging natural video temporal context substantially outperforms approaches focused on high-fidelity pixel reconstruction, revealing that temporal predictive structure—not reconstruction fidelity—is key to learning action-grounded visual representations. The resulting models exhibit superior generalization and robustness to visual perturbations across multiple robotic benchmarks, achieving a more favorable trade-off between visual fidelity and action prediction capability.

Abstract

Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-relevant structure in their latent spaces. We study this question through a unified probe-based evaluation across diverse encoder families, including image-only self-supervision, video pretraining with and without latent prediction, reconstruction-based autoencoders, diffusion models, and shortcut-forcing dynamics models. Using a common inverse-dynamics probing objective, we find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity: models with strong pixel decoding quality can exhibit near-zero action recoverability, while video-pretrained self-supervised encoders consistently achieve the best Pareto trade-off between visual fidelity and action prediction. Comparing V-JEPA and VideoMAE further shows that most gains arise from natural-video temporal context, with feature-level latent prediction providing a smaller additional benefit. These trends transfer across robotic benchmarks, though CALVIN reveals that static-environment tasks can partially mask the importance of temporal structure by allowing strong image priors to suffice. Finally, inverse-dynamics supervision substantially improves robustness to visual corruption, suggesting that action-aware objectives regularize latent geometry beyond clean-setting performance. Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.

Problem

• video world models
• action-relevant representations
• temporal pretraining
• latent structure
• inverse dynamics

Innovation

• temporal prediction
• action-relevant representation
• video world models
• inverse-dynamics probing
• self-supervised pretraining