Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents

2026-08-06

AI Summary

This study investigates dynamic selection of optimal observation modalities—text, pixels, or their fusion—for web automation tasks to balance success rate and computational cost. Through systematic evaluation across six observation modalities on diverse website–model combinations, the work proposes task-adaptive routing strategies, including confidence cascades, cost-aware tiering, and zero-cost rules, and uncovers a negative feedback loop between the scarcity of routing supervision signals and task difficulty. Experimental results demonstrate that a fixed, high-quality observation modality often outperforms complex routing schemes; furthermore, routing only failed tasks to the cheapest modality reduces costs by 9.5–30.6% without compromising success rates. The authors publicly release a complete measurement protocol and re-execution noise intervals, establishing clear boundaries for effective routing under cost constraints.

Abstract

Web agents observe a browser through text, pixels, or both, and the choice is usually fixed once for all tasks. We measure six observation modes across eight site-model combinations (cells) on VisualWebArena and WebArena and ask what choosing per task would buy. The modes are complementary: each solves tasks the others miss, they fail in structurally different ways, and the best choice reverses between task sets. The obvious prize, an oracle that picks a winning mode for every task, looks large but is inflated by run-to-run noise: rerunning the same mode on the same tasks changes 12-14% of outcomes, so a second run of a mode already in hand gains about as much as adding a new one. What survives is a cost bound: sending only the tasks no mode solves to the cheapest mode cuts cost by 9.5-30.6% in 8 of 8 cells at unchanged success. We then test five routing policies (picking the mode, deciding when to spend on the strong mode, a zero-cost rule read off the task text, a confidence cascade, and pooled cost tiers), and none robustly beats simply fixing one well-chosen mode; the one exception is a fragile result in our sparsest cell. The central obstruction is that routing supervision is produced at the agent's success rate: the weaker the agent, the fewer labels a router gets, exactly where routing would be most valuable. This limit belongs to today's agents rather than to routing itself. Label supply and routing opportunity rise together (correlation 0.95 across cells), so a stronger agent can overturn the result, and we report the rerun noise bands and the full measurement protocol.

Problem

• representation routing
• web agents
• observation modes
• task-specific routing
• supervision scarcity

Innovation

• representation routing
• web agents
• observation modes
• label scarcity
• cost-efficient inference