RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents

2026-10-04

AI Summary

This study addresses the error propagation problem in remote sensing agents caused by uncertainty in tool observations. To mitigate this issue, we propose a reliability assessment framework grounded in process evidence verification. The framework integrates Bayesian inference-based reliability modeling with task-tool priors constructed from offline feedback, enabling dynamic evaluation of observation quality through tool invocation monitoring and evidence chain analysis. Based on these assessments, it adaptively executes accept, supplement, or reject decisions. Experimental results demonstrate that the proposed approach significantly suppresses error propagation, effectively improves accuracy across multiple benchmark tasks, and reduces redundant tool invocations.

Abstract

Remote sensing agents rely on perception, measurement, and raster analysis tools to solve Earth observation tasks. We refer to their judgments and quantitative results about ground objects as tool observations. However, these observations are subject to substantial uncertainty and may be incorrect even when the tools execute successfully. When agents accept incorrect observations, the errors can propagate through subsequent reasoning and cause task failure. We analyze 1,229 execution trajectories across three remote sensing agent benchmarks. On each benchmark, at least 88.1% of tasks depend on tool observations. Among these tasks, at least 22.7% contain incorrect observations despite successful tool execution. These errors propagate to the final answer in at least 82.0% of affected tasks on each benchmark. To address this problem, we propose RSure-Agent, a framework for verifying tool observations and limiting error propagation. We introduce a verifiable observation protocol that requires tools to return process evidence for the agent to verify their observations. We also construct a task-tool reliability prior from offline task feedback. The prior summarizes each tool configuration's past performance across task types and provides a task-specific reference for verification. Using process evidence and this prior, RSure-Agent decides whether to accept an observation, request additional evidence, or reject it. We evaluate RSure-Agent on EarthBench, ThinkGeo, TerraLogic, and CHOICE-420. Across the three agent benchmarks, RSure-Agent reduces the error propagation rate by 21.3 to 25.9 percentage points relative to the base configuration with verification and the prior disabled. On CHOICE-420, it improves overall accuracy over direct answering by 5.71 percentage points on average across 11 backbone models. On EarthBench, it reduces the tool-call ratio by 25.9% relative to Earth-Agent.

Problem

• Remote Sensing Agents
• Tool Observations
• Error Propagation
• Uncertainty

Innovation

• Remote Sensing Agents
• Tool Observation Verification
• Error Propagation Mitigation
• Process Evidence
• Task-Tool Reliability Prior