When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving

2026-09-28

AI Summary

This study questions the validity of treating autonomous driving benchmark scores as optimization objectives. By decomposing the scoring pipeline into four stages, including execution and measurement, we employ controlled intervention experiments and closed-loop comparative analyses to reveal how overlooked behavioral discrepancies lead to score distortion. Furthermore, we find that closed-loop optimization gains are highly dependent on the execution interface. This work establishes a theoretical connection between behavior preservation and gain transferability, arguing that validating metric effectiveness requires jointly examining both the measurement semantics of scoring and the execution mechanisms underlying optimized behaviors. These findings offer a novel perspective for re-evaluating current autonomous driving assessment frameworks.

Abstract

Driving benchmark scores are increasingly used not only for evaluation but also as optimization targets. This raises a fundamental question: do score gains remain reliable evidence of driving improvement once the score itself is optimized? We address this question by examining how the scoring process responds to changes in driving behavior and whether the resulting gains persist under repeated execution and replanning. We decompose the process into execution, measurement, subscore mapping, and aggregation. Controlled interventions reveal substantial behavioral changes that receive little score response because distinctions are omitted, thresholded, or attenuated between requested and executed motion. Closed-loop comparisons further show that optimization gains can reverse when the execution interface changes, demonstrating their dependence on how requests are executed and returned as feedback. Together, these findings connect the behavioral distinctions preserved by a metric to the conditions under which its gains transfer. Metric validity under optimization therefore requires examining both what the scoring process measures and how the optimized behavior is executed.

Problem

• autonomous driving
• metric validity
• benchmark optimization
• driving evaluation
• score reliability

Innovation

• Metric Validity
• Autonomous Driving
• Benchmark Optimization
• Closed-loop Evaluation
• Score Decomposition