EchoRL: Reinforcement Learning via Rollout Echoing

2026-05-29

AI Summary

This work addresses a critical limitation in reinforcement learning from human feedback (RLHF) training: as successful trajectories converge, their reward variance approaches zero, leading to degenerate advantage estimates and vanishing policy gradients that hinder further performance gains. To mitigate this issue, the authors propose EchoClip, a novel method that leverages step-level entropy to identify high-information segments within otherwise degenerate trajectories and incorporates them as lightweight auxiliary supervision signals into the training objective, thereby reconstructing effective learning signals. EchoClip is compatible with diverse RLHF frameworks and large language model architectures, demonstrating consistent performance improvements across 10 benchmarks, 5 model variants, and 4 RLHF algorithms, all while introducing minimal computational overhead.

Abstract

Reinforcement Learning with Verifiable Rewards is an effective route for post-training to strengthen the reasoning capability of large language models. However, as training proceeds, the learning signal can collapse thus makes the training gain become marginal and ineffective. Specifically, a growing fraction of prompts' rollouts become advantage-degenerated: all the self-generated rollouts show verified-success, making the standard deviation over their rewards be zero; accordingly each rollout's advantage becomes degenerated (zero) as well. Given such rollouts' advantages, the policy-gradient for model optimization eventually vanishes, capping the training performance. We argue that some of these rollouts still contain valuable learning signals but unfortunately omitted with the existing RLVR methods. In this paper, inspired through analyzing the entropy pattern behind golden trajectories produced by external expert models, we propose EchoRL for better exploiting the advantage-degenerated rollouts to further improve the training performance. EchoRL is a lightweight module that first identifies an EchoClip from verified-success rollouts based on their step-level entropy values, and then feeds this clip back as an auxiliary supervision signal in the RL objective. Extensive experiments across 10 benchmarks, 5 LLM backbones, and 4 popular RLVR post-training methods demonstrate that EchoRL consistently improves RLVR post-training with minimal overhead.

Problem

• Reinforcement Learning
• Advantage Degeneration
• Reward Collapse
• Post-training
• Large Language Models

Innovation

• EchoRL
• Reinforcement Learning with Verifiable Rewards
• advantage degeneration
• entropy-based trajectory selection
• post-training for LLMs