AsyncWebRL: Efficient Multi-Step RL for Visual Web Agents

2026-06-03

AI Summary

This work addresses the inefficiency in training vision-language web agents via multi-step reinforcement learning, which often suffers from GPU underutilization and redundant trajectories. To mitigate these issues, the authors propose an asynchronous training framework that overlaps rollout execution, gradient updates, and policy refreshing. They further introduce constant normalization in place of trajectory-length normalization to effectively decouple the detrimental influence of failed trajectories on policy updates, thereby enhancing both training efficiency and policy conciseness. Integrated with a multi-step GRPO algorithm, an everlasting rollout pool, and lightweight screenshot processing, the proposed approach achieves a 48.7% success rate on the WebGym out-of-distribution test set—surpassing the previous state of the art by 5.8%, with notable improvements of 42% and 48% on medium and hard tasks, respectively.

Abstract

Training vision-language web agents with multi-step RL is compute-intensive, with two dominant forms of inefficiency: idle GPUs in synchronous RL, and trajectories that use more steps and tokens than necessary. We present AsyncWebRL, which addresses both. On the system side, an asynchronous design overlaps rollout, gradient update, and policy refresh across iterations, paired with two web-agent-specific adaptations, namely an everlasting rollout pool and lightweight screenshot handling, that together deliver up to a $2.9\times$ end-to-end training-throughput speedup over the previously fastest open synchronous pipeline (WebGym). On the algorithmic side, we identify the per-trajectory normalizer $1/|τ_i|$ in multi-step GRPO as the root cause of trajectory-level and token-level inefficiency: because failures are systematically longer than successes, it down-weights the negative gradient on failed tokens, so the policy keeps producing verbose memory schemas. Replacing $1/|τ_i|$ with a constant $1/k$ breaks this coupling, contracting trajectories while preserving aggregate success. Together, these contributions set a new open-source state of the art on the WebGym out-of-distribution test split (+5.8% relative over the 42.9% prior best), with the largest gains on the harder slices (+42% relative on Medium, +48% relative on Hard).

Problem

• multi-step RL
• visual web agents
• training inefficiency
• trajectory length
• token efficiency

Innovation

• asynchronous reinforcement learning
• multi-step RL
• vision-language web agents
• trajectory normalization
• training efficiency