MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold

2026-06-11

AI Summary

This work addresses the core challenges of generating interactive, temporally coherent 3D video worlds from a single narrow-field-of-view image—namely, limited field of view, geometric consistency, and camera controllability. The authors propose a decoupled framework that separates world construction from rendering: first, a topology-aware diffusion model synthesizes a 360° panoramic image; then, a persistent 3D Gaussian scaffold is built via panoramic geometry-aware residual prediction, which subsequently drives a Gaussian-conditioned video renderer to synthesize photorealistic videos along arbitrary camera trajectories. Innovatively combining the controllability of explicit 3D representations with the perceptual quality of generative models, the method introduces a bidirectional diffusion teacher–causal autoregressive student architecture enabling low-latency streaming rendering. Real-time interactive exploration at 8 FPS is achieved on a single RTX 4090, demonstrating for the first time the feasibility of single-image-driven interactive video world modeling.

Abstract

We present MoVerse, a real-time video world model that creates an interactively navigable scene from a single narrow-field-of-view image. This setting is challenging because the input observes only a small fraction of the environment, while interactive roaming requires a complete surrounding world, persistent geometry, controllable camera motion, and temporally coherent high-fidelity observations. MoVerse addresses this problem by separating world construction from observation rendering. It first expands the input into a gravity-aligned 360$^\circ$ panorama with topology-aware diffusion, closing the missing field of view before 3D reasoning. It then lifts the panorama into a persistent 3D Gaussian scaffold using panoramic geometry-aware residual prediction, yielding a dense and directly renderable spatial memory. Finally, a Gaussian-conditioned video renderer translates scaffold renderings along user-specified camera trajectories into photorealistic video. To make this renderer practical for interaction, we train a bidirectional diffusion teacher for high-quality conditional rendering and distill it into a causal autoregressive student for bounded-latency streaming. This design combines the controllability and long-range consistency of explicit 3D representations with the perceptual quality of generative video models. MoVerse supports real-time scene roaming at 8~FPS on a single NVIDIA RTX~4090 GPU, demonstrating a practical path toward single-image world creation with interactive video output.

Problem

• real-time video world modeling
• single-image reconstruction
• interactive scene roaming
• 360-degree panorama
• temporal coherence

Innovation

• Gaussian Scaffold
• Panoramic Diffusion
• World Modeling
• Conditional Video Rendering
• Real-Time Scene Roaming