Speculative Rollback Correction for Quality-Diverse Web Agent Imitation

2026-06-10

AI Summary

This work addresses the challenge of balancing intervention timing in imitation learning for web agents trained on expert trajectories: late interventions risk irreversible errors, while early ones induce policy dependence and local optima. To resolve this, the authors propose SRC, a branch-level imitation learning framework that employs speculative execution with a fixed horizon and a rollback mechanism. The teacher identifies the first harmful deviation and reverts to the preceding valid action prefix. High-quality, diverse trajectories are selected via a hard verifier and stored in a lightweight quality-diversity archive to supervise fine-tuning of the next action. Evaluated on WebArena-Infinity, SRC achieves superior performance over stepwise review using only 977 full trajectories and 9,183 action samples, reducing expert query costs while enhancing error recovery and preserving multiple solution paths.

Abstract

Training interactive web agents through imitation learning from expert trajectories has emerged as a highly effective approach. However, determining the optimal timing for expert intervention presents a critical challenge in this context. Delayed intervention often leads to the accumulation of early-stage errors, pushing the page state into an irrecoverable regime. Conversely, premature or excessive intervention causes the agent to become overly reliant on expert policies, trapping the model in local optima characterized by a single, rigid trajectory. We propose Speculative Rollback Correction (SRC), a branch-level imitation framework for resettable agent environments. Instead of requesting teacher labels at every visited state or correcting only after a completed trajectory, SRC uses fixed-horizon branch review: the student executes a short speculative segment before teacher review, and the teacher localizes the first harmful deviation only when local progress breaks. Rollback preserves useful prefixes, while successful rollouts are filtered by a hard verifier and retained in a lightweight quality-diversity archive. The resulting data supports next-action supervised fine-tuning on both localized corrections and verifier-passing trajectories. On WebArena-Infinity, SRC collects 977 verifier-passing trajectories and 9,183 next-action examples; fixed-horizon review improves the recovery-versus-query tradeoff over step-level review while retaining verifier-passing solution variants. Code is available at https://github.com/LongkunHao/SRC_gui_agent.

Problem

• imitation learning
• expert intervention
• web agents
• error recovery
• trajectory diversity

Innovation

• Speculative Rollback Correction
• imitation learning
• quality-diversity archive
• branch-level review
• web agent