SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

2026-09-17

AI Summary

论文解决了LLM在汽车安全分析中的可靠性问题，通过创建SAFARI基准来评估LLM辅助下的危害分析和风险评估能力。

Abstract

Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive Risk Inference), the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified industrial HARA cases and evaluates two coupled tasks: open-ended hazard analysis and standards-grounded risk assessment. To evaluate open-ended HARA artifacts, we propose the first reference-anchored LLM-as-a-judge protocol with high expert correlation. Experiments with nine frontier LLMs show that models often produce plausible hazard narratives but remain weak at ISO 26262 risk classification, with the best ASIL macro-F1 reaching only 0.261. Chain-of-Thought prompting provides limited benefit and often degrades categorical risk assessment. Error analysis further localizes major failures to scenario-critical context omissions during hazard generation and to controllability misjudgments during risk assessment, indicating where expert oversight should be concentrated. The dataset can be obtained from https://github.com/xixi47520-hash/HARA.

Problem

• Large language models
• Safety-critical engineering
• Automotive Hazard Analysis and Risk Assessment
• ISO 26262
• Reliability

Innovation

• SAFARI
• LLM-assisted HARA
• reference-anchored LLM-as-a-judge protocol
• ISO 26262 risk classification