Driving scenario generation and evaluation using a structured layer representation and foundational models

2025-11-03

AI Summary

To address the challenges of acquiring and evaluating rare, complex driving scenarios in autonomous driving development, this paper proposes a five-layer structured scenario generation and evaluation framework. Methodologically, it integrates foundational large language models with data augmentation strategies, introduces hierarchical sub-class definitions and fine-grained attribute descriptions to construct a domain-specific embedding space, and designs dual-dimensional evaluation metrics—diversity and originality—enabling end-to-end generation from structured scenario descriptions to synthetic videos. Contributions include: (1) the first structured scenario representation model supporting sub-class awareness and feature alignment; (2) a novel, quantifiable metric system for assessing the functional value of synthesized scenarios; and (3) open-sourced code and benchmark results demonstrating high fidelity of generated scenarios and empirical validity of the proposed evaluation metrics.

Technology Category

• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Rare and challenging driving scenarios are critical for autonomous vehicle development. Since they are difficult to encounter, simulating or generating them using generative models is a popular approach. Following previous efforts to structure driving scenario representations in a layer model, we propose a structured five-layer model to improve the evaluation and generation of rare scenarios. We use this model alongside large foundational models to generate new driving scenarios using a data augmentation strategy. Unlike previous representations, our structure introduces subclasses and characteristics for every agent of the scenario, allowing us to compare them using an embedding specific to our layer-model. We study and adapt two metrics to evaluate the relevance of a synthetic dataset in the context of a structured representation: the diversity score estimates how different the scenarios of a dataset are from one another, while the originality score calculates how similar a synthetic dataset is from a real reference set. This paper showcases both metrics in different generation setup, as well as a qualitative evaluation of synthetic videos generated from structured scenario descriptions. The code and extended results can be found at https://github.com/Valgiz/5LMSG.

Problem

• Generating rare driving scenarios using structured layer models
• Evaluating synthetic datasets with diversity and originality metrics
• Improving autonomous vehicle testing through structured scenario representation

Innovation

• Structured five-layer model for scenario representation
• Foundational models generate scenarios via data augmentation
• Embedding-based comparison with diversity and originality metrics