Problem
Research questions and friction points this paper is trying to address.
end-to-end driving
sim-to-real gap
policy perception
closed-loop simulation
visual fidelity
DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving
🤖 AI Summary
为解决仿真与现实视觉差异问题，提出DreamStream，使用基于模拟器的自回归视频模型生成保持策略相关特征的仿真环境。
📝 Abstract
Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene features a policy relies on to make decisions. Existing platforms, however, exhibit a sim-to-real visual gap that corrupts policy perception, undermining their ability to assess a policy's closed-loop decision-making. To this end, we propose DreamStream, a generative, closed-loop simulator that achieves policy-oriented fidelity using a simulator-grounded autoregressive video model. Our video model is distilled from a large pretrained video model via traffic layout guidance, varying visual appearance while preserving policy-relevant features such as scenario layout and the temporal consistency of dynamic objects. We further observe that perceptual metrics like FID misrank how well these features are preserved. To tackle this, we introduce FD$π$, a new multi-representation metric that measures the sim-to-real gap as the Fréchet distance over scene-context features from public E2E policies. Under FD$π$, DreamStream improves over the strongest prior closed-loop simulator by $1.6\times$ on nuScenes and $4.7\times$ on NAVSIM, and induces the least perturbation to policy's perceptual observability. Based on DreamStream, we construct Navhard-CL benchmark, which turns non-reactive real-world benchmark NAVSIM into interactive testing environments with adversarial driving behaviors and weather variations. This benchmark exposes many failure modes of driving policies, such as scorer bias and lack of recovery behaviors, that prior closed-loop benchmarks overlook. Code and data are available at https://github.com/VAIL-UCLA/DreamStream.
Innovation
Methods, ideas, or system contributions that make the work stand out.
policy-oriented fidelity
simulator-grounded autoregressive video model
FDπ
🔎 Similar Papers
No similar papers found.
