NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving

2026-03-09

AI Summary

This work addresses the challenge faced by current vision-language models (VLMs) in autonomous driving, where balancing high-level semantic reasoning with precise motion control often entails a trade-off between model scale and planning accuracy. To overcome this limitation, the authors propose a decoupled framework that explicitly separates reasoning from control for the first time: a large-scale vision-language model (Navigator) handles semantic understanding, while a lightweight, trainable driving module (Driver) executes motion planning. This design preserves strong semantic capabilities while significantly reducing training costs and yielding interpretable intermediate representations. Evaluated on the nuScenes end-to-end motion planning benchmark, the proposed approach outperforms existing large VLM-based baselines.

Technology Category

Computer Vision: Language and VisionPlanning, Routing, and Scheduling: Planning with Language ModelsNatural Language Processing: (Large) Language Models

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Large language models for searchUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-language models (VLMs) have emerged as a promising direction for end-to-end autonomous driving (AD) by jointly modeling visual observations, driving context, and language-based reasoning. However, existing VLM-based systems face a trade-off between high-level reasoning and motion planning: large models offer strong semantic understanding but are costly to adapt for precise control, whereas small VLM models can be fine-tuned efficiently but often exhibit weaker reasoning. We propose NaviDriveVLM, a decoupled framework that separates reasoning from action generation using a large-scale Navigator and a lightweight trainable Driver. This design preserves reasoning ability, reduces training cost, and provides an explicit interpretable intermediate representation for downstream planning. Experiments on the nuScenes benchmark show that NaviDriveVLM outperforms large VLM baselines in end-to-end motion planning.

Problem

• autonomous driving
• vision-language models
• motion planning
• high-level reasoning
• end-to-end learning

Innovation

• decoupled architecture
• vision-language model
• autonomous driving
• motion planning
• interpretable representation