PAN: A World Model for General, Interactable, and Long-Horizon World Simulation

2025-11-12

AI Summary

Current world models suffer from limited generalizability, lack of interactive capability, and insufficient long-horizon consistency, while video generation models lack causal control and action conditioning. This paper introduces GLP—a General, Interactive, and Long-horizon world model architecture—capable of high-fidelity, action-conditioned, video-level future state simulation in open domains. GLP unifies language instructions and visual dynamics within a shared latent space, integrating an autoregressive latent dynamics backbone driven by a large language model with a video diffusion decoder, trained on large-scale video–action pairs. Experiments demonstrate that GLP significantly outperforms existing methods on action-conditioned simulation, hundred-frame-long horizon prediction, and simulation-based reasoning tasks, exhibiting strong generalization and practical utility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

A world model enables an intelligent agent to imagine, predict, and reason about how the world evolves in response to its actions, and accordingly to plan and strategize. While recent video generation models produce realistic visual sequences, they typically operate in the prompt-to-full-video manner without causal control, interactivity, or long-horizon consistency required for purposeful reasoning. Existing world modeling efforts, on the other hand, often focus on restricted domains (e.g., physical, game, or 3D-scene dynamics) with limited depth and controllability, and struggle to generalize across diverse environments and interaction formats. In this work, we introduce PAN, a general, interactable, and long-horizon world model that predicts future world states through high-quality video simulation conditioned on history and natural language actions. PAN employs the Generative Latent Prediction (GLP) architecture that combines an autoregressive latent dynamics backbone based on a large language model (LLM), which grounds simulation in extensive text-based knowledge and enables conditioning on language-specified actions, with a video diffusion decoder that reconstructs perceptually detailed and temporally coherent visual observations, to achieve a unification between latent space reasoning (imagination) and realizable world dynamics (reality). Trained on large-scale video-action pairs spanning diverse domains, PAN supports open-domain, action-conditioned simulation with coherent, long-term dynamics. Extensive experiments show that PAN achieves strong performance in action-conditioned world simulation, long-horizon forecasting, and simulative reasoning compared to other video generators and world models, taking a step towards general world models that enable predictive simulation of future world states for reasoning and acting.

Problem

• Existing video models lack causal control and long-horizon consistency for reasoning
• Current world models are restricted to specific domains with limited generalization
• There is no unified system combining latent reasoning with realistic visual dynamics

Innovation

• Autoregressive latent dynamics with LLM backbone
• Video diffusion decoder for visual reconstruction
• Generative Latent Prediction architecture unification