HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

2025-01-24

AI Summary

Existing Driving World Models (DWMs) support only scene generation, lacking geometric, semantic, and dynamic understanding of the surrounding environment—limiting prediction and decision-making in autonomous driving. This paper introduces HERMES, a Unified Driving World Model that jointly models 3D scene understanding and future scene generation for the first time. Its core innovations include: (1) an integrated BEV architecture unifying perception and generation; (2) a large language model–driven world query mechanism that injects causal prior knowledge into BEV features via causal attention; and (3) cross-task context sharing with end-to-end joint optimization. Evaluated on nuScenes and OmniDrive-nuScenes, HERMES achieves state-of-the-art performance: reducing generation error by 32.4% and improving the CIDEr understanding metric by 8.0%. The code and models are publicly available.

Technology Category

Planning, Routing, and Scheduling: Planning with Language ModelsHumans and AI: Human-Aware Planning and Behavior PredictionMachine Learning: Large Multimodal Models (LMMs)

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model (LLM), enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at https://github.com/LMD0311/HERMES.

Problem

• Autonomous Vehicles
• Driving World Models
• Environmental Understanding

Innovation

• HERMES model
• Bird's eye view
• World Query