End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation

2026-07-20

AI Summary

Existing methods for generating traffic scenarios struggle to simultaneously achieve photorealistic fidelity and fine-grained controllability, particularly in safety-critical interactions. This work proposes E2E-CDiff, a novel framework that introduces, for the first time, an end-to-end joint diffusion model over states and actions for this task. Given front-view image observations, E2E-CDiff concurrently generates surrounding vehicles’ future trajectories and low-level control commands. The approach unifies semantic behavioral control and physical constraints through a conditional diffusion model, joint denoising, and differentiable guidance mechanisms—including speed modulation, drivable area constraints, and collision guidance. Experiments on Bench2Drive demonstrate that the method outperforms reinforcement learning and imitation learning baselines, striking a favorable balance between realism and controllability. Notably, its collision-guided variant effectively elicits challenging interactions for autonomous driving systems and exhibits promising potential as an end-to-end planner.

Abstract

Generating closed-loop traffic scenarios that are both realistic and controllable is crucial for evaluating autonomous driving systems, especially under rare safety-critical interactions. Existing learning-based methods often struggle to balance controllability and realism, offering either limited fine-grained control over traffic behavior or controllable scenarios at the expense of behavioral plausibility. This paper presents E2E-CDiff, an end-to-end conditional diffusion framework for controllable and realistic scenario generation. Conditioned on front-view visual observations, E2E-CDiff jointly denoises future motion states and executable low-level controls for route-interacting background vehicles. This unified state-action generation mitigates the planning-control mismatch in conventional two-stage trajectory-then-controller pipelines. Differentiable guidance further regulates speed, enforces drivable-area compliance, and supports collision-avoidance or collision-seeking behaviors, enabling both naturalistic and safety-critical scenario generation. Experiments on Bench2Drive show that E2E-CDiff achieves a favorable controllability-realism trade-off compared with representative reinforcement- and imitation-learning baselines, while its collision-guided variant induces challenging interactions across multiple autonomous driving systems. E2E-CDiff also performs competitively as a learning-based ego planner, demonstrating the generality of end-to-end state-action diffusion.

Problem

• traffic scenario generation
• realism
• controllability
• autonomous driving
• safety-critical interactions

Innovation

• conditional diffusion
• end-to-end generation
• controllable realism
• state-action joint modeling
• differentiable guidance