VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving

2026-06-10

AI Summary

Current evaluations of vision-language-action (VLA) models focus narrowly on driving trajectory quality, overlooking the relevance, consistency, and causality between chain-of-thought (CoT) reasoning and executed actions. This work proposes VLADriveBench, a framework that systematically integrates observational metrics—such as referentiality, hallucination, contradiction, and action alignment—with CoT intervention protocols and visual saliency analysis to assess the causal relationship between CoT and driving actions from complementary perspectives. Experiments on three state-of-the-art models reveal a significant divergence between observational metrics and intervention outcomes: while ORION achieves the highest action alignment score, its CoT appears incidental; in contrast, Alpamayo v1.5, despite lower scores, demonstrates strong causal CoT, thereby exposing critical limitations in existing evaluation paradigms.

Abstract

Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.

Problem

• Vision-language-action
• Chain-of-Thought
• Autonomous Driving
• CoT-action relationship
• Benchmarking

Innovation

• Chain-of-Thought
• Vision-Language-Action
• Causal Evaluation
• Autonomous Driving
• Intervention Protocol