VideoWorld: Exploring Knowledge Learning from Unlabeled Videos

2025-01-16

AI Summary

This work investigates a self-supervised learning paradigm driven solely by unlabeled video data, aiming to master complex tasks—such as Go and robotic control—without reward signals or explicit search strategies. We propose VideoWorld: an end-to-end autoregressive video generation framework featuring a novel Latent Dynamics Model (LDM) that explicitly captures spatiotemporal causal transitions in visual streams, integrated with unsupervised spatiotemporal representation learning and video-conditioned policy distillation. Our key contribution is the first empirical demonstration that raw video streams inherently encode sufficient semantic and dynamical knowledge to support rule understanding, reasoning, and planning. On Video-GoBench, a 300M-parameter VideoWorld model achieves professional 5-dan level performance—without search or reward. On CALVIN and RLBench, it approaches oracle-level performance and exhibits strong cross-environment generalization.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This work explores whether a deep generative model can learn complex knowledge solely from visual input, in contrast to the prevalent focus on text-based models like large language models (LLMs). We develop VideoWorld, an auto-regressive video generation model trained on unlabeled video data, and test its knowledge acquisition abilities in video-based Go and robotic control tasks. Our experiments reveal two key findings: (1) video-only training provides sufficient information for learning knowledge, including rules, reasoning and planning capabilities, and (2) the representation of visual change is crucial for knowledge acquisition. To improve both the efficiency and efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional level in the Video-GoBench with just a 300-million-parameter model, without relying on search algorithms or reward mechanisms typical in reinforcement learning. In robotic tasks, VideoWorld effectively learns diverse control operations and generalizes across environments, approaching the performance of oracle models in CALVIN and RLBench. This study opens new avenues for knowledge acquisition from visual data, with all code, data, and models open-sourced for further research.

Problem

• Self-supervised Learning
• Unlabeled Video Data
• Complex Task Mastery

Innovation

• VideoWorld model
• LDM component
• unsupervised learning