Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents

2026-08-04

AI Summary

This study addresses decision-making biases in hybrid GUI-MCP agents that lead to low tool utilization (only 23.9%) and unstable performance, particularly pronounced in non-reasoning models. The authors identify a “adoption gap” stemming from insufficient task-specific training and propose a context compression mechanism that reduces input tokens by approximately one-third while improving accuracy—effectively closing the performance gap between rich and simplified observation modes for the first time. Through an end-to-end optimization framework incorporating multi-round reinforcement learning probes, dense tool-use rewards, image history truncation, and retraining strategies, the compressed agent achieves a task success rate of 37.8% at 53% of the original input cost, up from 33.0%, and entirely eliminates observation-mode discrepancies on a pre-registered evaluation subset.

Abstract

Hybrid computer-use agents can act through screenshots or call text tools. We find that having a tool available does not settle which way the effect goes. Under one identical GUI-MCP harness on the OSWorld-MCP benchmark (309 tasks), the same MCP tools improve a reasoning model by +4.0pp and degrade a non-reasoning model by -5.9pp (5 runs each, both beyond 2 SE). What separates the two is tool-decision behavior. The non-reasoning policy ignores, misnames, or falsely terminates around tools. The reasoning model avoids these failures, yet still calls a tool on only 55/309 tasks, 23.9% of the tool-reachable ones. We call this shortfall the adoption gap. Both levels of the problem share one cause: the model already has a cheaper route and is never trained to take it. Multi-turn RL probes that cause. At the action level, a dense tool bonus raises spreadsheet adoption 0.03 -> 0.33 and carries into greedy decoding, but held-out accuracy does not follow. Behavior is steerable; competence is not. The bottleneck lies in tool-call semantics. At the context level, a successful tool call often makes the next screenshot redundant. Dropping it and halving image history cuts input tokens by about a third, at a small accuracy cost. Retraining under the same observation rule removes that cost. The compressed agent then reaches 37.8% against 33.0% for the uncompressed operating point, at 53% of the input cost, and closes the rich-lean gap on a pre-registered degraded subset to zero. Tools help when the model chooses and integrates them, and current hybrid agents leave many such choices unused.

Problem

• tool use
• adoption gap
• hybrid agents
• multimodal context
• GUI-MCP

Innovation

• tool adoption gap
• hybrid GUI-MCP agents
• multimodal context compression
• tool-call semantics
• reinforcement learning probing