WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA

2026-08-02

AI Summary

This work addresses the high latency and exposure bias inherent in autoregressive decoding of vision–language–action (VLA) models for autonomous driving, as well as the limited multitask reasoning capability of diffusion-based policies. To overcome these challenges, the authors propose a three-stage hierarchical distillation framework that efficiently transfers knowledge from a pretrained autoregressive generalist model to a parallel diffusion architecture. This approach uniquely aligns causal and bidirectional attention mechanisms and integrates block-wise adaptive distillation, cross-scale knowledge transfer, and system-level optimizations such as FlashInfer and CUDA Graphs. The resulting model preserves strong multitask semantic understanding while achieving substantially improved inference efficiency. Experiments demonstrate that the method matches the performance of autoregressive baselines across driving comprehension, perception, and planning tasks, with a 2.8× speedup in decoding and an overall end-to-end acceleration of 15.1×.

Abstract

Vision-Language-Action (VLA) models have emerged as a prominent paradigm for end-to-end autonomous driving; however, their efficient deployment is severely constrained by high computational latency and exposure bias arising from sequential autoregressive decoding. Conversely, while specialized diffusion policies enable low-latency, parallel execution, training them from scratch typically yields narrow, single-task architectures that lack holistic visual-linguistic reasoning. Successfully transforming pre-trained autoregressive generalists into parallel diffusion models could combine multi-task cognitive intelligence with execution efficiency, yet this transition presents a formidable architectural challenge due to mismatched attention patterns (causal versus bidirectional) and divergent optimization objectives. To bridge this divide, we introduce WAM-Diff2, a multi-task discrete diffusion VLA framework powered by a three-stage hierarchical distillation strategy. By structuring the architectural shift through progressive block-wise adaptation, block-wise distillation, and model-wise cross-scale distillation, WAM-Diff2 preserves the underlying semantic foundations of the base model while accelerating inference. Extensive evaluations across driving understanding, perception, and planning benchmarks demonstrate that WAM-Diff2 effectively mitigates exposure bias and achieves performance parity with autoregressive baselines. Crucially, the autoregressive-to-diffusion transition yields a 2.8x decoding speedup, which scales to an ultimate 15.1x acceleration when combined with system-level optimizations including FlashInfer and CUDA Graphs.

Problem

• Vision-Language-Action
• autoregressive-to-diffusion
• exposure bias
• computational latency
• architectural mismatch

Innovation

• hierarchical distillation
• autoregressive-to-diffusion
• vision-language-action
• diffusion policy
• efficient autonomous driving