map evaluation

This work addresses the poor reproducibility and inconsistent benchmarking of real-time object detection models. We establish a standardized training and multi-GPU inference evaluation framework built upon MMDetection, systematically reproducing state-of-the-art models—including DETR, RTMDet, ViTDet, and YOLOv7—on MS COCO 2017. Our methodology ensures end-to-end reproducible configurations, hardware-agnostic (multi-GPU) joint evaluation of accuracy and latency, and strict alignment with original training protocols and hyperparameters. Key contributions are: (1) demonstrating the superior accuracy–latency trade-off of anchor-free detectors (e.g., RTMDet, YOLOX); (2) revealing widespread reproducibility challenges—RTMDet and YOLOv7 achieve original performance, whereas DETR and ViTDet fall short; and (3) quantitatively confirming a strong negative correlation between accuracy and inference speed, along with significant degradation in inference efficiency of pre-trained models under resource-constrained conditions.

To address perception distortion induced by sensor data compression and virtualization in autonomous driving, this paper proposes a four-step quantitative framework: (1) constructing paired distorted image sets; (2) measuring image fidelity degradation using LPIPS, SSIM, and PSNR; (3) evaluating task-level performance degradation—specifically mAP reduction and increased localization error—on object detection models (YOLO, Faster R-CNN); and (4) establishing statistical correlations between image quality metrics and downstream task performance. This work is the first to quantitatively characterize the relationship between image distortion magnitude and model robustness degradation in autonomous driving perception tasks. Results show that LPIPS exhibits the strongest correlation with performance degradation (Spearman’s ρ > 0.92), significantly outperforming SSIM and PSNR. The framework establishes a reproducible, interpretable, data-quality-driven paradigm for robustness validation of machine learning systems in safety-critical perception applications.

Detecting small, occluded, and multi-scale objects—such as traffic signs, African wildlife, and small vessels—remains challenging for real-time object detectors. Method: This work systematically benchmarks YOLOv3 through YOLOv11 across these three demanding datasets under a unified evaluation framework, measuring detection accuracy (mAP₅₀₋₉₅), latency, computational cost (GFLOPs), and model size (MB). Contribution/Results: It presents the first comprehensive empirical analysis of the YOLOv11 series, identifying YOLOv11m as the Pareto-optimal variant: achieving mAP₅₀₋₉₅ of 0.795, 0.810, and 0.325 on the respective datasets; attaining 2.4 ms average inference latency; requiring only 38.8 MB storage and 67.6 GFLOPs. Compared to YOLOv9 (accurate but inefficient) and YOLOv10 (fast but weak on occlusion), YOLOv11m establishes a new accuracy-efficiency trade-off benchmark. These findings provide evidence-based guidance for detector selection, lightweight design, and small-object detection optimization.

To address the suboptimal performance of general-purpose vision-language models (VLMs) on geospatial tasks—such as environmental monitoring and disaster response—this work introduces GeoBench, the first VLM benchmark dedicated to remote sensing understanding. GeoBench comprises over 10,000 human-verified, multi-source remote sensing instructions spanning six core tasks: scene understanding, fine-grained classification, object counting, localization, segmentation, and cross-temporal analysis. It systematically evaluates VLM capabilities on geospatially unique challenges, including detection of tiny objects, large-scale counting, and change identification. Experimental results reveal a significant capability gap: the state-of-the-art model LLaVA-OneVision achieves only 41.7% accuracy on multiple-choice tasks—substantially below its performance in general-domain benchmarks. GeoBench is publicly released to serve as a standardized evaluation platform for advancing geospatial AI research.

This work proposes a Membership Inference Test (MINT) framework tailored for object detection models to address the risks of training data memorization and privacy leakage. By analyzing activation patterns in intermediate layers, the method integrates an object detector, an embedding extractor, and a customized MINT module to effectively determine whether a given input sample was part of the training set. Experiments on three public datasets—comprising over 174K images—demonstrate that the proposed approach achieves membership inference accuracy of 70%–80%. The study further identifies key factors influencing inference performance, such as the depth of the detection module’s input layer, thereby significantly enhancing the applicability and auditability of membership inference in complex vision tasks.

Existing object detection models lack intuitive, fine-grained methods for performance comparison, making it difficult to uncover their shared and distinct failure modes in recognizing ground-truth labels. To address this, this work proposes Differences in Detection (DnD), a novel approach that introduces a structured set-partitioning mechanism based on standard matching algorithms. By decomposing model behaviors into intersections, differences, and co-missed sets, DnD enables direct pairwise comparison and integrates the TIDE error taxonomy to construct an interpretable confusion matrix. Moving beyond conventional metrics like mAP and isolated error statistics, the method clearly delineates shared versus unique errors, thereby guiding interpretability techniques—such as ODAM—to prioritize critical samples that reveal meaningful discrepancies between detectors.

This work addresses the limitation of current vision models that predominantly rely on ISP-processed RGB images, thereby discarding sensor-native information beneficial for machine reasoning and hindering efficient object detection and description under low-bit quantization. To bridge this gap, we present RAWDet-7, the first systematic benchmark for joint object detection and description on low-bit quantized RAW images. RAWDet-7 comprises approximately 25k training and 7.6k test images across seven object categories, offering diverse real-world RAW data captured across varying cameras, lighting conditions, and scenes, along with aligned object-level semantic descriptions. The benchmark supports 4/6/8-bit quantization studies and adheres to MS-COCO and LVIS annotation standards, providing a standardized platform to evaluate detection accuracy, description quality, detail preservation, and generalization—thereby advancing the effective utilization of raw sensor data in machine vision.

Current geospatial foundation models (GeoFMs) lack standardized evaluation protocols across tasks and data characteristics, hindering fair performance comparison and application-specific model selection. To address this, we propose the first comprehensive benchmarking framework for GeoFMs, introducing a novel “capability grouping” paradigm that systematically characterizes model capabilities along key data dimensions—spatial resolution, spectral band count, and temporal granularity. The framework covers five core remote sensing tasks: classification, semantic/instance segmentation, object detection, and regression, integrating 19 open-source Earth observation (EO) datasets. It provides a standardized, modular, and multimodal-compatible evaluation protocol. We publicly release open-licensed code, benchmark data, and a dynamic leaderboard. Experimental results reveal no universally dominant model: natural-image pre-trained models excel in high-resolution tasks, whereas domain-specific GeoFMs outperform in multispectral modeling. This work advances standardization and reproducibility in EO AI research.

This study addresses the lack of systematic validation regarding the effectiveness of large models for small object detection in resource-constrained Earth observation scenarios. Using rooftop photovoltaic detection in Madagascar as a case study, the authors systematically evaluate the impact of model scale, training data volume, and input resolution on detection efficiency. They report the first observation of an “efficiency inversion” phenomenon: the smallest model, YOLO11N, achieves both the highest mAP50 (0.617) and the greatest computational efficiency per unit cost. Moreover, high-resolution configurations of smaller models exhibit Pareto dominance across all deployment settings. Experimental results show that YOLO11N is 24 times more efficient than YOLO11X, increasing input resolution yields a 120% efficiency gain, and augmenting training data under low-resolution settings provides negligible benefits.

In object detection, existing online data selection methods suffer from architectural complexity and domain shift, hindering effective sample value assessment. This paper proposes DetGain—the first framework to introduce marginal contribution modeling into online data selection for object detection. DetGain dynamically estimates each sample’s marginal gain in mean Average Precision (mAP) by quantifying its perturbation effect on the global mAP and integrating teacher-student prediction discrepancies. It requires no detector architecture modification, relying solely on prediction quality evaluation, global score distribution modeling, and teacher-student divergence analysis—ensuring low intrusiveness and strong generalizability. Extensive experiments across multiple detectors on COCO demonstrate that DetGain significantly accelerates convergence and improves final accuracy, exhibits robustness to noisy or low-quality data, and seamlessly synergizes with knowledge distillation for further performance gains.