SUV: Future Scene Understanding as Video Generation for End-to-End Driving

2026-08-03

AI Summary

This work addresses the limitations of existing end-to-end autonomous driving approaches, which struggle to jointly model multimodal future scene representations—such as appearance, semantics, depth, and instance dynamics—and rely on task-specific prediction heads that hinder scalability. To overcome these challenges, the paper proposes a unified framework that formulates future scene understanding as a video generation task, leveraging a pretrained video foundation model to simultaneously predict multidimensional future states. A joint video-action attention mechanism directly generates the ego-vehicle trajectory, eliminating the need for multiple specialized heads and enabling shared representations with structured supervision. Using only a single front-facing monocular camera, the method achieves 91.0 EPDMS on navtest and 36.9 on navhard of NAVSIM-v2, and attains 7.94 RFS on WOD-E2E, significantly outperforming current state-of-the-art methods.

Abstract

End-to-end driving requires a coherent understanding of future scenes, yet existing methods model these scenes using task-specific heads and output formats, with limited scalability. Can video generation instead provide a shared predictor? We introduce SUV, a unified end-to-end driving framework that casts future Scene Understanding as Video generation using a pretrained video foundation model. SUV models future appearance, semantics, relative depth, and instance-level dynamics as video streams with a shared video expert, without stream-specific visual prediction heads. Through joint video-action attention, the action expert attends to the latent representations of all future streams and generates the ego trajectory. Experiments show that SUV directly predicts all four future streams, while controlled ablations show that structured future supervision and direct future-stream access yield higher trajectory planning scores. With only a single front camera and no candidate-trajectory selection, SUV outperforms a broad set of recent state-of-the-art methods on both NAVSIM-v2 splits, achieving 91.0 EPDMS on navtest and 36.9 on navhard. On the long-tail WOD-E2E benchmark, SUV achieves a competitive RFS of 7.94.

Problem

• end-to-end driving
• future scene understanding
• video generation
• scalability
• scene modeling

Innovation

• video generation
• end-to-end driving
• future scene understanding
• foundation model
• unified framework