Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks

2026-07-22

AI Summary

This work addresses the credit assignment trap in long-horizon agent tasks, where sparse rewards lead to undersampling of critical state-changing actions and over-reinforcement of ineffective behaviors due to sampling imbalance. To mitigate this, the authors propose ProGPO, a method that introduces a progress signal based on first-visit observation coverage within entirely failed trajectories. This signal enables a progress-conditioned advantage estimation mechanism that treats state coverage as an intrinsic reward, thereby guiding the policy to prioritize exploration of novel states. Implemented within a population-based policy optimization framework, ProGPO is evaluated using Qwen2.5-1.5B/7B-Instruct models in ALFWorld and WebShop environments, demonstrating significant performance gains over existing baselines on both overall and challenging tasks, effectively alleviating the credit assignment trap and enhancing exploration efficiency.

Abstract

Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon tasks, this comparison can suffer from a sampling imbalance: repeated or low-effect actions dominate the high-probability region of the policy while useful state-changing actions remain under-sampled. This imbalance produces many all-failed rollout groups, where outcome rewards provide no direction for correcting the policy. Together, these effects can form a self-reinforcing credit trap: failure-dominated sampling yields no outcome-based correction, allowing repeated low-effect actions to persist. To break this loop, we propose Progress-conditioned Group Policy Optimization (ProGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward. Specifically, within such groups, ProGPO assigns higher relative advantages to trajectories or steps that visit more new states since reaching new observations is a prerequisite for task success. Experiments on two challenging agentic benchmarks, ALFWorld and WebShop with Qwen2.5-1.5/7B-Instruct, show that ProGPO consistently improves over group-based baselines, with particularly large gains on hard tasks.

Problem

• long-horizon tasks
• sampling imbalance
• sparse rewards
• credit assignment
• policy optimization

Innovation

• Progress-conditioned
• Group Policy Optimization
• Long-Horizon Agentic Tasks
• Sparse Rewards
• Exploration Guidance