evaluate detection metrics

Traditional benchmarks provide only aggregate scores, offering insufficient evidence to support reliable deployment decisions and thereby creating a disconnect between evaluation and action. To address this gap, this work proposes a “deployment-completeness” benchmarking framework, introducing novel metrics—evidence fibers, completeness curves, and certifiable proportions—alongside a systematic audit methodology comprising evidence fiber analysis, response ranking intervals, conformal coverage evaluation, and a certify-then-acquire decision pipeline. Empirical evaluation on benchmarks such as Tox21, Matbench, and JARVIS reveals that conventional approaches suffer a drastic drop in channel coverage to 10.07% under real-world deployment conditions. In contrast, the proposed method reduces error-driven deployment decisions to 0.027% on Tox21 and 0.128% on JARVIS, substantially enhancing deployment reliability.

Current clinically deployed AI systems for pulmonary nodule detection lack post-deployment validation against variations in local CT acquisition parameters, potentially leading to substantial performance deviations from benchmark expectations. This work proposes a reproducible validation framework suitable for resource-constrained settings that does not require proprietary scanner data. The approach leverages physics-based image-domain simulations: Gaussian noise models reduced radiation dose, while z-axis moving averages emulate slice thickness alterations. Sensitivity is evaluated using a 15-mm matching criterion. Experiments reveal that slice thickness exerts a far greater impact on model performance than noise—increasing thickness to 5 mm reduces the sensitivity of MONAI RetinaNet from a baseline of 45.2% to 26.2% (a 42% relative drop), whereas a 50% dose reduction only mildly decreases sensitivity to 42.1%. Notably, some cases already fail at baseline, underscoring the critical need for post-deployment validation.

This work addresses the lack of systematic and rigorous performance benchmarking methodologies in programming language research, which has undermined the credibility of evaluation results. To remedy this, the paper introduces a closed-loop methodology—Measure-Explain-Test-Improve—that establishes, for the first time, a structured and reproducible workflow for performance assessment in the field. Integrating systematic experimental design, performance metric analysis, result interpretation, and iterative refinement, the approach emphasizes theoretical grounding and practical rigor at every stage. Its key contribution lies in enabling even researchers with limited empirical experience to conduct reliable and methodologically sound performance evaluations, thereby significantly enhancing the scientific validity and reproducibility of performance analysis in programming language research.

Existing object detection models lack intuitive, fine-grained methods for performance comparison, making it difficult to uncover their shared and distinct failure modes in recognizing ground-truth labels. To address this, this work proposes Differences in Detection (DnD), a novel approach that introduces a structured set-partitioning mechanism based on standard matching algorithms. By decomposing model behaviors into intersections, differences, and co-missed sets, DnD enables direct pairwise comparison and integrates the TIDE error taxonomy to construct an interpretable confusion matrix. Moving beyond conventional metrics like mAP and isolated error statistics, the method clearly delineates shared versus unique errors, thereby guiding interpretability techniques—such as ODAM—to prioritize critical samples that reveal meaningful discrepancies between detectors.

This work addresses the poor reproducibility and inconsistent benchmarking of real-time object detection models. We establish a standardized training and multi-GPU inference evaluation framework built upon MMDetection, systematically reproducing state-of-the-art models—including DETR, RTMDet, ViTDet, and YOLOv7—on MS COCO 2017. Our methodology ensures end-to-end reproducible configurations, hardware-agnostic (multi-GPU) joint evaluation of accuracy and latency, and strict alignment with original training protocols and hyperparameters. Key contributions are: (1) demonstrating the superior accuracy–latency trade-off of anchor-free detectors (e.g., RTMDet, YOLOX); (2) revealing widespread reproducibility challenges—RTMDet and YOLOv7 achieve original performance, whereas DETR and ViTDet fall short; and (3) quantitatively confirming a strong negative correlation between accuracy and inference speed, along with significant degradation in inference efficiency of pre-trained models under resource-constrained conditions.

This work addresses the challenge of efficiently translating mission intent into structured, machine-readable execution outcomes on in-orbit satellites under stringent communication and power constraints. The authors present a hardware-in-the-loop intelligent agent system deployed on a commercial ARM heterogeneous edge SoC, which orchestrates a local large language model and a YOLO-style oriented object detection module via FastAPI within a browser-based workspace to enable task orchestration and real-time state feedback. The study establishes, for the first time, an observable boundary for satellite edge intelligent agent orchestration, supports FAIR1M metadata-compliant output, and provides a fully reproducible experimental package. Empirical results demonstrate 100% success across 20 trials for two FAIR1M workloads, achieving end-to-end latencies of 29.35 s and 60.94 s, with the detection module consuming less than 3% of total runtime, average CPU utilization at 20.6%, and NPU utilization reaching 100%.

This work addresses the limitations of existing deep detector–based traffic perception methods, which rely on pretrained models and struggle in scenarios lacking annotations or involving open-set vehicle categories. The authors propose a lightweight, purely geometric, and category-agnostic traffic perception framework that extracts moving objects via background subtraction and thresholding, then counts vehicles using geometric rules applied along virtual induction lines—eliminating the need for object detection models, training data, or trajectory tracking. A self-calibrating pre-calibration mechanism is further introduced to recover lane geometry and count lane occupancy boundaries through patch statistics, while also estimating average per-vehicle speed at no additional computational cost. Evaluated on four video sequences, the method achieves counting accuracies of 83.3%–100%, with 91% accuracy in real-world deployment, substantially outperforming a patch-tracking baseline (37.5%) under comparable computational constraints.