LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving

2025-07-08

AI Summary

Urban autonomous driving faces core challenges in complex and edge-case scenarios, including insufficient semantic understanding, biased intent recognition, and decision-making logic diverging from human driving norms. To address these, we propose a dual-rate decision architecture integrating large language models (LLMs): a high-frequency end-to-end module ensures real-time control, while a low-frequency semantic module leverages LLMs for multimodal perception fusion and chain-of-thought reasoning—significantly enhancing situational awareness and robustness in unconventional scenarios. Crucially, we introduce LLMs into the planning closed loop, enabling human-aligned, robust decisions when conventional planners fail. Furthermore, we enhance end-to-end training via high-definition map integration and imitation learning. Evaluated on the CARLA Leaderboard V1 benchmark, our method achieves a score of 71 and a route completion rate of 93%, outperforming state-of-the-art approaches.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

A principal barrier to large-scale deployment of urban autonomous driving systems lies in the prevalence of complex scenarios and edge cases. Existing systems fail to effectively interpret semantic information within traffic contexts and discern intentions of other participants, consequently generating decisions misaligned with skilled drivers' reasoning patterns. We present LeAD, a dual-rate autonomous driving architecture integrating imitation learning-based end-to-end (E2E) frameworks with large language model (LLM) augmentation. The high-frequency E2E subsystem maintains real-time perception-planning-control cycles, while the low-frequency LLM module enhances scenario comprehension through multi-modal perception fusion with HD maps and derives optimal decisions via chain-of-thought (CoT) reasoning when baseline planners encounter capability limitations. Our experimental evaluation in the CARLA Simulator demonstrates LeAD's superior handling of unconventional scenarios, achieving 71 points on Leaderboard V1 benchmark, with a route completion of 93%.

Problem

• Handling complex urban driving scenarios and edge cases
• Interpreting semantic traffic information and participant intentions
• Aligning decisions with skilled driver reasoning patterns

Innovation

• Dual-rate architecture with E2E and LLM
• LLM enhances scenario comprehension via multi-modal fusion
• Chain-of-thought reasoning optimizes decision-making