PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning

2026-07-23

AI Summary

This work addresses the challenge of sparse trajectory information in long-horizon reinforcement learning for large language model (LLM) agents, where weak policies often fail repeatedly, hindering effective policy optimization. The authors propose a policy-centric training paradigm that dynamically models skills as evolving scaffolds aligned with policy development. Specifically, during inference, the framework adaptively provides guidance through evidence card generation, task-specific evaluation, and context-aware adjustment mechanisms, gradually reducing reliance on external support as the agent’s capabilities improve—thus balancing guided assistance with growing autonomy. Integrated with standard RLVR optimization, this approach outperforms strong baselines by up to 18.6% on ALFWorld and WebShop benchmarks, achieves competitive performance across seven retrieval-augmented question-answering tasks, and reduces prompt usage by 32.1%.

Abstract

In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, Pats, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. On ALFWorld and WebShop, Pats improves over strong baselines by up to 18.6%. Across seven search-augmented QA benchmarks, it remains competitive while using 32.1% fewer prompt tokens than the baseline.

Problem

• long-horizon reinforcement learning
• weak policies
• uninformative rollouts
• policy optimization
• LLM agents

Innovation

• policy-aware scaffolding
• dynamic training support
• evidence cards
• adaptive context guidance
• agentic reinforcement learning