What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs

2026-05-31

AI Summary

Current vision-language models (VLMs) for autonomous driving suffer from sparse validation coverage within their operational design domain (ODD), leading to unreliable empirical failure rates. To address this, this work proposes SliceScorer—a scoring mechanism—and SliceNav, a validation pipeline that jointly incorporates exposure frequency priors and proximity-based failure propagation priors to proactively identify and recommend high-risk, under-tested scenario slices. The approach leverages large language models (LLMs) to orchestrate an interpretable, deterministic, and end-to-end validation workflow. For the first time, it integrates deterministic risk scoring with LLM-driven validation scheduling. Experiments on WiseAD, DriveMM, and Cosmos-Reason2-2B demonstrate that SliceNav more efficiently uncovers high-risk coverage gaps and yields greater recommendation diversity compared to existing methods, with ablation studies confirming the contribution of each component.

Abstract

Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs), yet verification remains sparse: many slices are missing, making empirical failure rates unreliable. We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation that combines (i) an exposure-based coverage prior to prioritize rare, under-tested regions, and (ii) a neighbor-failure prior that propagates risk from similar tested conditions. SliceScorer is deliberately simple - interpretable, auditable, and conservative - properties essential for safety-critical validation. For stress testing beyond the declared ODD, we embed SliceScorer within SliceNav, an LLM-orchestrated verification pipeline where the model interprets developer queries to select relevant operators (triage, scoring, acquisition, evaluation) and vocabulary extensions, composing verification workflows while keeping all scoring deterministic and auditable. Experiments on three driving VLMs (WiseAD, DriveMM, Cosmos-Reason2-2B) show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods while maintaining diverse recommendations across the condition space. Ablations confirm both scoring components contribute, and qualitative analysis demonstrates end-to-end workflows from developer query to targeted evaluation.

Problem

• coverage gap
• driving VLMs
• Operational Design Domain
• verification
• slice discovery

Innovation

• coverage gap discovery
• interpretable scoring
• driving VLMs
• deterministic verification
• LLM-orchestrated validation