A Survey on the Application of Large Language Models in Scenario-Based Testing of Automated Driving Systems

2025-05-22

AI Summary

Scenario-based testing of autonomous driving systems (ADS) urgently requires efficient methodologies, yet no systematic survey exists on the application of large language models (LLMs) in this domain. Method: This paper presents the first comprehensive, ADS-scenario-testing–focused survey of LLMs, proposing a five-stage application framework—test preparation, scenario generation, test execution, result analysis, and knowledge consolidation—derived from literature review and cross-domain technology mapping. We formally model LLMs’ core capability boundaries and identify five fundamental open challenges. Contribution/Results: We construct a structured LLM application taxonomy for ADS testing and release an actively maintained, open-source knowledge repository on GitHub, integrating state-of-the-art research, toolchains, and industrial case studies. This work provides both a reusable methodological foundation and collaborative R&D infrastructure for academia and industry.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The safety and reliability of Automated Driving Systems (ADSs) must be validated prior to large-scale deployment. Among existing validation approaches, scenario-based testing has been regarded as a promising method to improve testing efficiency and reduce associated costs. Recently, the emergence of Large Language Models (LLMs) has introduced new opportunities to reinforce this approach. While an increasing number of studies have explored the use of LLMs in the field of automated driving, a dedicated review focusing on their application within scenario-based testing remains absent. This survey addresses this gap by systematically categorizing the roles played by LLMs across various phased of scenario-based testing, drawing from both academic research and industrial practice. In addition, key characteristics of LLMs and corresponding usage strategies are comprehensively summarized. The paper concludes by outlining five open challenges and potential research directions. To support ongoing research efforts, a continuously updated repository of recent advancements and relevant open-source tools is made available at: https://github.com/ftgTUGraz/LLM4ADSTest.

Problem

• Validating safety and reliability of Automated Driving Systems (ADSs)
• Exploring LLMs' role in scenario-based testing for ADSs
• Addressing lack of dedicated reviews on LLMs in ADS testing

Innovation

• LLMs enhance scenario-based testing efficiency
• Systematic categorization of LLMs roles
• Comprehensive summary of LLMs usage strategies