Talk2DM: Enabling Natural Language Querying and Commonsense Reasoning for Vehicle-Road-Cloud Integrated Dynamic Maps with Large Language Models

2026-02-12

AI Summary

This study addresses the lack of natural language interaction capabilities in existing vehicle-road-cloud integrated dynamic maps, which hinders efficient human-machine collaboration. To overcome this limitation, the authors propose Talk2DM, a novel module that leverages a newly developed VRCsim simulation framework and the accompanying VRC-QA question-answering dataset. The approach introduces a Chain-of-Prompting (CoP) mechanism that seamlessly integrates handcrafted rules with commonsense knowledge from large language models (LLMs), enabling natural language querying and reasoning over dynamic maps. Designed for plug-and-play compatibility, Talk2DM supports multiple LLMs—including Qwen3-8B, Gemma3-27B, and GPT-oss—and achieves over 93% query accuracy on the VRC-QA benchmark with an average response time of only 2–5 seconds, substantially enhancing both interaction efficiency and practical utility.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Dynamic maps (DM) serve as the fundamental information infrastructure for vehicle-road-cloud (VRC) cooperative autonomous driving in China and Japan. By providing comprehensive traffic scene representations, DM overcome the limitations of standalone autonomous driving systems (ADS), such as physical occlusions. Although DM-enhanced ADS have been successfully deployed in real-world applications in Japan, existing DM systems still lack a natural-language-supported (NLS) human interface, which could substantially enhance human-DM interaction. To address this gap, this paper introduces VRCsim, a VRC cooperative perception (CP) simulation framework designed to generate streaming VRC-CP data. Based on VRCsim, we construct a question-answering data set, VRC-QA, focused on spatial querying and reasoning in mixed-traffic scenes. Building upon VRCsim and VRC-QA, we further propose Talk2DM, a plug-and-play module that extends VRC-DM systems with NLS querying and commonsense reasoning capabilities. Talk2DM is built upon a novel chain-of-prompt (CoP) mechanism that progressively integrates human-defined rules with the commonsense knowledge of large language models (LLMs). Experiments on VRC-QA show that Talk2DM can seamlessly switch across different LLMs while maintaining high NLS query accuracy, demonstrating strong generalization capability. Although larger models tend to achieve higher accuracy, they incur significant efficiency degradation. Our results reveal that Talk2DM, powered by Qwen3:8B, Gemma3:27B, and GPT-oss models, achieves over 93\% NLS query accuracy with an average response time of only 2-5 seconds, indicating strong practical potential.

Problem

• Dynamic Maps
• Natural Language Querying
• Vehicle-Road-Cloud Integration
• Commonsense Reasoning
• Human-DM Interaction

Innovation

• Chain-of-Prompt
• Natural Language Querying
• Vehicle-Road-Cloud Integration
• Dynamic Maps
• Commonsense Reasoning