🤖 AI Summary
A systematic taxonomy and survey of world models bridging 2D perception to 3D cognition remains absent. Method: We propose a dual-axis classification framework—“3D representation advancement” and “world knowledge integration”—to systematically characterize the technical evolution of 3D-cognitive world models. Our approach unifies neural radiance fields (NeRF), 3D generative modeling, spatial reasoning networks, physics simulation, and multimodal knowledge integration into a coherent conceptual framework for 3D spatial cognition. Contribution/Results: We formally identify three core capabilities—3D scene generation, spatial reasoning, and embodied interaction—and clarify their interdependencies. Furthermore, we pinpoint critical challenges including data scarcity, limited modeling generalizability, and real-time deployment constraints. This work establishes a foundational theoretical basis and provides a principled technical roadmap toward developing generalizable, robust 3D-cognitive systems.
📝 Abstract
World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.