Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles, Part II: Assessment Results

2026-10-06

AI Summary

This study addresses the critical need for independent third-party behavioral safety assessments to foster public trust in the large-scale deployment of autonomous vehicles, a domain where existing standards remain insufficient. To bridge this gap, this work proposes a systematic third-party testing framework that leverages the Mcity proving ground and high-fidelity simulation environments to comprehensively evaluate Autoware.Universe, an open-source Level 4 autonomous driving system. The proposed framework effectively uncovers previously unidentified unsafe scenarios, thereby addressing the absence of established evaluation standards. Empirical results demonstrate that the evaluated system possesses only six of fourteen assessed capabilities and exhibits an accident rate approximately one thousand times higher than that of human drivers. These findings underscore the indispensable role of independent behavioral safety assessment in ensuring the safe deployment of autonomous driving systems.

Abstract

Third-party evaluations of autonomous vehicle (AV) safety can play a vital role in improving public acceptance, building consumer confidence, and establishing effective safety standards. In Part I of this study, we propose a dedicated third-party testing initiative for systematically evaluating AV behavioral safety. In this paper, we validate our proposed framework using Autoware.Universe, an open-source Level 4 Automated Driving System (ADS), tested both in simulated environments and on the physical test track at the University of Michigan's Mcity Testing Facility. The results indicate that Autoware.Universe possesses 6 out of 14 behavioral competencies and exhibited a crash rate of 3.01x10^-3 crashes per mile, approximately 1,000 times higher than the average human driver crash rate. During the tests, we also uncovered a number of unknown unsafe scenarios for Autoware.Universe. These findings underscore the necessity of behavioral safety evaluations for improving AV safety performance prior to widespread public deployment.

Problem

• Autonomous Vehicles
• Behavioral Safety Assessment
• Third-party Evaluation
• Safety Performance

Innovation

• Behavioral Safety Assessment
• Autonomous Vehicles
• Third-party Evaluation
• Autoware.Universe
• Unsafe Scenarios