ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systems

2025-03-26

AI Summary

Large multimodal models (LMMs) exhibit insufficient generalization in autonomous driving due to misinterpretation of traffic knowledge, complexity of real-world road conditions, and diversity of vehicle states. Method: We propose a novel knowledge editing paradigm that avoids full retraining, and introduce ADS-Edit—the first multimodal knowledge editing benchmark tailored for autonomous driving. ADS-Edit comprises error patterns mined from real-world driving logs, diverse road scenarios, multi-vehicle dynamic states, and multimodal inputs. We further design a cross-modal consistency evaluation mechanism and knowledge-editing prompting strategies to enable precise, efficient correction of LMMs’ traffic cognition. Contribution/Results: Experiments demonstrate substantial improvements in critical capabilities—including traffic rule comprehension and anomaly response—while significantly enhancing robustness and generalization under dynamic driving conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements in Large Multimodal Models (LMMs) have shown promise in Autonomous Driving Systems (ADS). However, their direct application to ADS is hindered by challenges such as misunderstanding of traffic knowledge, complex road conditions, and diverse states of vehicle. To address these challenges, we propose the use of Knowledge Editing, which enables targeted modifications to a model's behavior without the need for full retraining. Meanwhile, we introduce ADS-Edit, a multimodal knowledge editing dataset specifically designed for ADS, which includes various real-world scenarios, multiple data types, and comprehensive evaluation metrics. We conduct comprehensive experiments and derive several interesting conclusions. We hope that our work will contribute to the further advancement of knowledge editing applications in the field of autonomous driving. Code and data are available in https://github.com/zjunlp/EasyEdit.

Problem

• Address misunderstanding of traffic knowledge in LMMs
• Handle complex road conditions in autonomous driving
• Manage diverse vehicle states with knowledge editing

Innovation

• Knowledge Editing for targeted model behavior modification
• Multimodal dataset ADS-Edit for autonomous driving scenarios
• Comprehensive evaluation metrics for diverse real-world conditions