geospatial accuracy assessment

Current interactive agent benchmarks often rely on superficial signals to determine task success, making it difficult to reliably verify whether the intended goal state has been genuinely achieved, thereby distorting evaluation outcomes. This work proposes a general-purpose augmentation layer that requires no modifications to tasks, agents, or evaluators. It introduces predefined validation artifacts and executes a locked checklist, explicitly handling uncertainty through three evidence labels: Evidence Pass, Fail, and Unknown. By constructing an evidence-backed scoring boundary mechanism, the approach isolates uncertain cases from conventional binary success rates, substantially enhancing evaluation transparency and reliability. Experiments across five benchmarks—including ANDROIDWORLD and AGENTDOJO—demonstrate that the framework effectively identifies diverse real-world failure modes and uncovers biases obscured by traditional evaluation protocols.

Existing evaluation methods struggle to effectively assess the dynamic execution capabilities of large language models (LLMs) in complex, multi-step geospatial analysis tasks and lack support for runtime feedback and the multimodal nature of spatial outputs. To address this, this work introduces a dynamic, interactive benchmark tailored for tool-augmented GIS agents, encompassing 117 atomic GIS tools and 53 representative tasks. It proposes the Parameter Execution Accuracy (PEA) metric, a “Last-Try Alignment” strategy, and a vision-language model–based mechanism for validating spatial outputs. Furthermore, the study designs a Plan-and-React agent architecture that decouples global planning from local reactive execution. Experimental results demonstrate that this architecture significantly outperforms baseline approaches across seven mainstream LLMs, achieving both logical rigor in multi-step reasoning and robustness in error recovery.

Satellite-derived 3D geospatial products—including point clouds, digital surface models (DSMs), and 3D meshes—lack a standardized, quantitative resolution metric. Method: This study proposes the first automated resolution assessment framework tailored to heterogeneous satellite-based 3D products. It integrates 3D metric analysis, robust point cloud registration with error mapping, surface geometric fidelity modeling, and a multi-scale resolution characterization workflow, enabling fully automatic comparison against high-accuracy airborne LiDAR reference data. Results: Validated across diverse satellite 3D datasets of varying sources and quality levels, the framework delivers quantitative resolution metrics, enables cross-source comparability, and supports accuracy闭环 validation for large-scale 3D scene modeling. It significantly enhances objectivity, reproducibility, and performance traceability of 3D geospatial information products.

Existing evaluation benchmarks for remote sensing agents lack realistic, application-oriented assessment of tool-use capabilities. Method: We introduce the first tool-augmented benchmark for remote sensing agents, covering seven real-world tasks—including urban planning and disaster assessment—that require multi-step tool invocation and spatial reasoning over satellite/aerial imagery. Our framework systematically evaluates tool-use proficiency through structured task design, human-in-the-loop query construction, and a dual-dimensional evaluation metric (“step-by-step execution” + “final answer”). Built upon the ReAct paradigm, it integrates remote sensing understanding, geospatial tool invocation, and multi-step planning. Results: Evaluated on 436 tasks, our benchmark reveals significant disparities among models (e.g., GPT-4o, Qwen2.5) in tool accuracy and planning consistency. All code and data are publicly released, establishing a foundational benchmark for embodied intelligence in remote sensing.

Existing evaluation of generative AI is hindered by the scarcity of high-quality benchmarks, whose manual construction is costly and time-consuming. Method: We propose the first automated benchmark construction framework powered by collaborative large language model (LLM) agents, decomposing benchmark creation into four sequential stages—planning, generation, verification, and evaluation—integrating task decomposition, agent coordination, human-in-the-loop feedback, and explicit constraint-satisfaction assessment. Contribution/Results: The framework significantly enhances data diversity and metric reliability. Leveraging it, we construct the first high-quality benchmark specifically targeting planning and constraint-satisfaction capabilities in text generation. We systematically evaluate seven state-of-the-art models, uncovering shared failure modes and fine-grained capability disparities. Our work establishes a scalable, reproducible paradigm for evaluating generative AI capabilities, advancing both benchmark methodology and empirical analysis.

This study addresses a critical gap in existing tool-calling evaluation benchmarks: the lack of validation of the evaluators themselves, which risks conflating assessment artifacts with agents’ true capabilities. Through a systematic audit of four prominent benchmarks—BFCL v4, τ2-Bench, LiveMCPBench, and MCP-Atlas—the authors conduct expert review of 496 tasks, replicate experiments, and perform trajectory-level analysis, revealing an 18.5% disagreement rate between automated evaluators and human judgment. Notably, LiveMCPBench exhibits a score variance of up to 18.9 percentage points upon re-evaluation, sufficient to overturn leaderboard rankings. To address these issues, the work introduces the first unified taxonomy of tool-calling evaluation failures, advocates for distinct measurement of tool invocation, task completion, and result verification, and releases Tool-Veritas—a configurable benchmark—and Harness Lab, an open-source evaluation platform.

This study addresses the lack of standardized, dynamic-aware approaches for assessing FAIR (Findable, Accessible, Interoperable, Reusable) compliance in geospatial datasets, particularly concerning evolving content and dataset-specific identifiers. To this end, the authors propose the first multi-agent collaborative framework for FAIR evaluation, integrating structured metadata extraction with 13 large language model–based evaluators, each aligned with a specific FAIR sub-principle. A critic agent is introduced to verify the quality and consistency of supporting evidence, enabling fine-grained scoring, automated evidence citation, and closed-loop reassessment. Experiments on 50 datasets yield average scores of 79.7%, 70.4%, 45.3%, and 72.0% across the four FAIR dimensions. Incorporating the critic agent improves inter-evaluator consistency by 18% to 89%, achieving a Fleiss’ kappa of 0.71, at a cost of approximately $0.054 per evaluation, thereby significantly enhancing transparency, consistency, and auditability.

Current agent benchmarks often yield misleading evaluation scores due to invalid protocols, primarily stemming from reward hacking or assessment vulnerabilities. This work presents the first systematic formalization of “protocol validity” and introduces Mislead Gap—a quantitative metric—and HackDetect, a posterior auditing framework. By integrating trajectory auditing, exposure point identification, and intent-exploitation score comparison, the framework uniformly detects and quantifies the impact of reward hacking. Empirical analysis across 15 benchmarks and 2,385 agent trajectories reveals that 66.7%–67.0% of evaluations exhibit exposure to or active engagement in reward hacking, inflating scores by 0.45–1.00. These findings demonstrate that prevailing benchmarks generally fail to validate agents’ true capabilities.

Current computer-using agent (CUA) benchmarks rely on fragile scripted evaluators that frequently produce erroneous failure judgments, obscuring true performance bottlenecks. This work proposes the first reliability-focused evaluation framework encompassing the entire pipeline—from task construction and trajectory observation to scoring and reporting—and introduces a three-tier failure diagnosis taxonomy. Through manual auditing and attribution analysis of 150 publicly reported failure trajectories, we find that 15.3% of failure labels are incorrect, with 10.7% stemming from evaluator misjudgment and 4.7% arising from task design flaws. Building on these insights, we derive phased design principles for long-horizon CUA evaluation, substantially improving assessment accuracy and interpretability.

This work addresses the limitations of existing evaluation methods that focus solely on final outcomes, which fail to distinguish reliable reasoning from accidental success or diagnose process-level flaws in long-horizon tasks. To this end, we propose ClawTrack, a dual-dimensional evaluation framework that jointly assesses task completion (Task Score) and reasoning process quality (Process Score). Spanning 320 tasks across eight domains, ClawTrack introduces fine-grained, stepwise scoring along four dimensions, enabling the first interpretable, process-level evaluation of autonomous agent reasoning trajectories. Our Process Grader combines rule-based logic with large language models, incorporating 12,541 task-specific scoring criteria and integrating over 25 deterministic simulation environments. Validation across 21 models and more than 16,000 trials demonstrates that process scores effectively attribute success or failure, filter out spurious successes, and—when used to select high-quality reasoning trajectories—significantly boost performance across model scales, with consistent results across different evaluator models.