PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning

2026-08-02

AI Summary

This work addresses the challenge of optimizing end-to-end autonomous driving systems under sole supervision from trajectory outputs, where weak gradient signals hinder effective joint learning across perception, prediction, and planning modules. To overcome this limitation, the authors propose a probabilistic deep supervision framework that models intermediate latent variables as reparameterizable distributions and directly regularizes these representations using ground-truth trajectory data. Central to this approach is a novel variational supervision mechanism based on the evidence lower bound (ELBO), which strengthens motion planning learning by explicitly aligning latent representations with observed trajectories. The method further reveals that the performance gains previously attributed to vision-language model supervision actually stem from implicit alignment with ground-truth trajectories. Evaluated on the nuScenes dataset, the proposed approach reduces planning L2 error by 8% and collision rate by 3% with negligible additional computational overhead.

Abstract

End-to-end autonomous driving (E2E AD) systems integrate perception, prediction, and planning into a single differentiable architecture. While these models show great promise, their standard training often relies on output-only supervision, which can lead to weak gradients for the hidden layers of increasingly complex models. Recent works have integrated vision-language model (VLM) supervision for latent features to address this, yielding substantial empirical gains, yet leaving the underlying theoretical mechanisms poorly understood. Our investigation into this methodology reveals that the resulting performance gains stem not from VLM reasoning capabilities, as previously assumed, but rather from the latent connections forged between the E2E AD model and ground-truth (GT) data during training. Building on this insight, we propose a probabilistic deep supervision framework that regularizes intermediate latent representations directly from GT data. By treating model latents as reparameterizable distributions, we optimize the architecture via the Evidence Lower Bound (ELBO). Our evaluations conducted on the nuScenes dataset demonstrate that supervising trajectory-related latents with future GT paths consistently improves planning performance. Using identical training data and E2E architectures, our method achieves an 8% reduction in planning L2 error and a 3% decrease in collision rates compared to competitive vectorized baselines, all while incurring negligible computational overhead.

Problem

• end-to-end autonomous driving
• latent supervision
• weak gradients
• motion planning
• ground-truth supervision

Innovation

• probabilistic latent supervision
• end-to-end autonomous driving
• deep supervision
• Evidence Lower Bound (ELBO)
• motion planning