A Comprehensive LLM-powered Framework for Driving Intelligence Evaluation

2025-03-07

AI Summary

Existing evaluation methods for autonomous driving intelligence in complex traffic scenarios lack systematicity and interpretability. Method: This paper proposes the first hierarchical intelligence assessment framework tailored to naturalistic driving behavior. It integrates naturalistic driving data collection, semantic interviews with drivers and passengers for annotation, and instruction-tuned large language models (LLMs) to construct a multidimensional, cognitively interpretable quantitative evaluation system. Crucially, it establishes the first LLM-driven assessment paradigm grounded in human driving cognition, enabling a paradigm shift from rule-based to semantics-driven evaluation. Results: In CARLA simulations, the proposed intelligence score achieves high agreement with human expert assessments (Spearman ρ = 0.92) and supports fine-grained attribution analysis. The open-sourced dataset and evaluation framework have been widely adopted in the research community.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Evaluation methods for autonomous driving are crucial for algorithm optimization. However, due to the complexity of driving intelligence, there is currently no comprehensive evaluation method for the level of autonomous driving intelligence. In this paper, we propose an evaluation framework for driving behavior intelligence in complex traffic environments, aiming to fill this gap. We constructed a natural language evaluation dataset of human professional drivers and passengers through naturalistic driving experiments and post-driving behavior evaluation interviews. Based on this dataset, we developed an LLM-powered driving evaluation framework. The effectiveness of this framework was validated through simulated experiments in the CARLA urban traffic simulator and further corroborated by human assessment. Our research provides valuable insights for evaluating and designing more intelligent, human-like autonomous driving agents. The implementation details of the framework and detailed information about the dataset can be found at Github.

Problem

• Lack of comprehensive autonomous driving intelligence evaluation methods.
• Need for driving behavior assessment in complex traffic environments.
• Development of an LLM-powered framework for driving intelligence evaluation.

Innovation

• LLM-powered framework for driving intelligence evaluation
• Natural language dataset from driving experiments
• Validation using CARLA simulator and human assessment