ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

2026-08-03

AI Summary

Existing tool-use benchmarks rely on static semantics, making it difficult to evaluate an agent’s ability to autonomously infer and adapt to unknown systems through interaction without prior knowledge. This work proposes ScrambleToolBench—an interactive terminal-based benchmark that removes semantic cues and introduces dynamic environmental perturbations, such as mapping drift and stochastic action failures, alongside a curriculum of sequential tasks, thereby forcing agents to rely solely on trial-and-error for behavioral reasoning. For the first time in tool-use evaluation, this benchmark isolates and explicitly targets behavioral reasoning capabilities, exposing fundamental limitations in current agents: a lack of deductive reasoning under structural changes and an overreliance on exhaustive search. Experiments show that even state-of-the-art language models struggle to adapt efficiently to structural shifts; while persistent memory mitigates error accumulation, it does not enhance reasoning efficiency.

Abstract

To operate robustly in open-world environments, autonomous agents should be able to infer the behavior of unfamiliar systems through interaction alone, even in the absence of documentation. However, existing tool-use benchmarks expose semantic tool schemas in static environments, allowing agents to rely on prior knowledge rather than autonomous discovery. To address this limitation, we introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning. By removing semantic cues and enforcing a continuous task curriculum, the benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction. The benchmark further introduces dynamic challenges, including mapping drift, stochastic action failures, and temporal execution windows, to evaluate whether agents can revise and adapt their hypotheses as the environment changes. Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation. When faced with structural changes such as mapping drift, agents fail to use deductive strategies such as cycle tracing, and instead exhibit belief inertia or fall back to exhaustive search. Increasing test-time reasoning only amplifies this expensive brute-force search rather than enabling deductive recovery. While equipping agents with persistent memory reduces compounding errors, they remain unable to efficiently infer structural changes, highlighting a gap in current agent reasoning.

Problem

• tool-use benchmark
• behavioral reasoning
• open-world environments
• autonomous discovery
• dynamic adaptation

Innovation

• behavioral reasoning
• tool-use benchmark
• mapping drift
• trial-and-error interaction
• deductive reasoning