Rethink 3D Object Detection from Physical World

2025-06-30

AI Summary

Current 3D detection evaluation over-relies on mAP and latency as isolated metrics, neglecting the accuracy–latency trade-off, cross-hardware performance variability, and—critically—the real-world impact of detection errors on motion planning safety. To address this, we propose a novel evaluation framework introducing latency-aware AP (L-AP) and planning-aware AP (P-AP), the first metrics to explicitly quantify the coupling among detection models, hardware accelerators, and downstream planning risk. Leveraging the nuPlan dataset and latency-aware hyperparameter optimization (L-HPO), we conduct joint evaluation across diverse hardware platforms and point-cloud detectors. Experiments reveal that increased point-cloud density does not necessarily improve planning safety; our method achieves state-of-the-art real-time detection performance while providing quantifiable, hardware-aware optimization guidance for autonomous driving co-design of software and hardware.

Technology Category

Machine Learning: Hardware-aware MLHumans and AI: Human-Aware Planning and Behavior PredictionPlanning, Routing, and Scheduling: Deterministic Planning

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsSystems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterizationSecurity and Privacy: Large-scale security measurements

Abstract

High-accuracy and low-latency 3D object detection is essential for autonomous driving systems. While previous studies on 3D object detection often evaluate performance based on mean average precision (mAP) and latency, they typically fail to address the trade-off between speed and accuracy, such as 60.0 mAP at 100 ms vs 61.0 mAP at 500 ms. A quantitative assessment of the trade-offs between different hardware devices and accelerators remains unexplored, despite being critical for real-time applications. Furthermore, they overlook the impact on collision avoidance in motion planning, for example, 60.0 mAP leading to safer motion planning or 61.0 mAP leading to high-risk motion planning. In this paper, we introduce latency-aware AP (L-AP) and planning-aware AP (P-AP) as new metrics, which consider the physical world such as the concept of time and physical constraints, offering a more comprehensive evaluation for real-time 3D object detection. We demonstrate the effectiveness of our metrics for the entire autonomous driving system using nuPlan dataset, and evaluate 3D object detection models accounting for hardware differences and accelerators. We also develop a state-of-the-art performance model for real-time 3D object detection through latency-aware hyperparameter optimization (L-HPO) using our metrics. Additionally, we quantitatively demonstrate that the assumption "the more point clouds, the better the recognition performance" is incorrect for real-time applications and optimize both hardware and model selection using our metrics.

Problem

• Address speed-accuracy trade-off in 3D object detection
• Evaluate hardware impact on real-time detection performance
• Assess detection's effect on motion planning safety

Innovation

• Introduces latency-aware AP (L-AP) metric
• Develops planning-aware AP (P-AP) metric
• Optimizes with latency-aware hyperparameter tuning