Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving

2026-07-31

AI Summary

This work addresses the issue of conditional policy collapse in end-to-end autonomous driving, where vision-language models often disregard navigation instructions. To mitigate this, the authors propose a single-pass, classifier-free guidance mechanism that introduces a novel class-center-based latent space shifting strategy. This approach transforms instance-level guidance into more stable category-level guidance by leveraging mean-shift principles and precomputed instruction-specific latent centroids, enabling efficient alignment between predicted trajectories and language commands within a single forward pass. Experimental results demonstrate that the method significantly enhances both instruction-following fidelity and overall driving performance on the Bench2Drive and nuScenes benchmarks, while simultaneously reducing inference latency by approximately 50%.

Abstract

Vision-language models (VLMs) have recently emerged as a promising paradigm for end-to-end autonomous driving, enabling agents to map multimodal inputs and high-level navigation instructions directly to executable trajectories. However, in practice, these models exhibit a persistent command-following gap: predicted trajectories often show weak sensitivity to navigation commands, resulting in incorrect behavior at critical decision points. We identify this issue as a form of conditional policy collapse, where regression-based training under multimodal trajectory distributions encourages the model to rely on dominant visual priors while marginalizing the language-conditioned signal. To address this issue, we introduce a principled formulation of classifier-free guidance (CFG) for regression-based vision-language driving. We show that CFG can be interpreted as isolating the instruction-induced residual in the action space by contrasting conditional and unconditional predictions, thereby explicitly amplifying the effect of the navigation command at inference time. However, a standard two-pass CFG introduces prohibitive latency for real-time control and produces noisy instance-level guidance directions. Building on a mean-shift interpretation of CFG, we propose Latent-Centroid Steering (LCS), a single-pass guidance mechanism that replaces instance-level residuals with class-level latent shifts. By projecting conditional representations toward precomputed command-specific centroids, LCS performs class-level latent steering based on cluster geometry that is both more stable and computationally efficient. We demonstrate that LCS reduces inference latency by approximately 50% while achieving stronger command adherence and improved driving performance on both closed-loop (Bench2Drive) and open-loop (nuScenes) benchmarks. Code will be released.

Problem

• command-following gap
• conditional policy collapse
• vision-language models
• autonomous driving
• navigation instructions

Innovation

• Latent-Centroid Steering
• Classifier-Free Guidance
• Vision-Language Driving
• Command-Aligned Autonomy
• Single-Pass Inference