Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving

2025-12-13

AI Summary

Current trajectory prediction evaluation over-relies on post-hoc metrics (e.g., ADE/FDE), neglecting the actual impact of predictions on autonomous vehicle (AV) decision-making safety—particularly failing to capture the coupling between prediction diversity and scenario criticality. This work proposes a scenario-driven, dual-dimensional dynamic evaluation paradigm: it introduces an adaptive weighting mechanism grounded in quantitative modeling of scenario criticality, enabling closed-loop simulation scoring that jointly considers prediction accuracy and diversity. Furthermore, it establishes, for the first time, a rigorous validation framework linking predictor performance to real-world AV driving behavior. Evaluated on a real-data closed-loop benchmark, the proposed metric achieves a 42% improvement in correlation with AV safety performance over conventional metrics, significantly outperforming them and precisely identifying predictors that most effectively support conservative, safety-critical decision-making.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Calibration & Uncertainty Quantification
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

Being able to anticipate the motion of surrounding agents is essential for the safe operation of autonomous driving systems in dynamic situations. While various methods have been proposed for trajectory prediction, the current evaluation practices still rely on error-based metrics (e.g., ADE, FDE), which reveal the accuracy from a post-hoc view but ignore the actual effect the predictor brings to the self-driving vehicles (SDVs), especially in complex interactive scenarios: a high-quality predictor not only chases accuracy, but should also captures all possible directions a neighbor agent might move, to support the SDVs' cautious decision-making. Given that the existing metrics hardly account for this standard, in our work, we propose a comprehensive pipeline that adaptively evaluates the predictor's performance by two dimensions: accuracy and diversity. Based on the criticality of the driving scenario, these two dimensions are dynamically combined and result in a final score for the predictor's performance. Extensive experiments on a closed-loop benchmark using real-world datasets show that our pipeline yields a more reasonable evaluation than traditional metrics by better reflecting the correlation of the predictors' evaluation with the autonomous vehicles' driving performance. This evaluation pipeline shows a robust way to select a predictor that potentially contributes most to the SDV's driving performance.

Problem

• Evaluates trajectory predictors beyond error-based metrics
• Assesses predictor impact on autonomous vehicle decision-making
• Proposes accuracy-diversity scoring for scenario-driven performance

Innovation

• Scenario-driven evaluation pipeline for trajectory predictors
• Dynamic combination of accuracy and diversity metrics
• Correlation with autonomous vehicle driving performance