Orbis 2: A Hierarchical World Model for Driving

2026-07-17

AI Summary

Existing driving world models struggle to simultaneously achieve high perceptual fidelity and robust high-level semantic understanding, limiting their applicability in real-world scenarios. This work proposes a two-tier hierarchical world model that jointly models spatiotemporal dynamics and semantics: a high-level predictor forecasts coarse-grained scene structures over long horizons, while a low-level generator synthesizes high-fidelity image details conditioned on the high-level outputs. The approach integrates hierarchical temporal modeling, diffusion-based pretraining with teacher forcing, and autoregressive fine-tuning to enhance representational richness and inference stability. Evaluated on standard benchmarks for driving world models, the proposed method achieves state-of-the-art performance in long-horizon generation quality, counterfactual responsiveness, and the effectiveness of its internal representations.

Abstract

Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/

Problem

• world models
• hierarchical representation
• spatial reasoning
• semantic understanding
• driving

Innovation

• hierarchical world model
• diffusion forcing
• teacher forcing
• spatial reasoning
• driving simulation