Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

2026-07-30

AI Summary

This work addresses the challenges of convergence difficulty and ambiguous optimization directions in privileged self-distillation within multi-turn agent scenarios, which stem from exposure bias. To mitigate these issues, the paper introduces contrastive learning into this framework for the first time, proposing an entropy-guided grouped contrastive self-distillation mechanism. This approach leverages prediction entropy to differentiate positive and negative samples, thereby preserving fine-grained and reliable optimization signals. The method effectively alleviates exposure bias and significantly enhances training stability and generalization capability in long-horizon interactions. Empirical results demonstrate substantial performance gains over existing reinforcement learning and self-distillation approaches across 13 challenging benchmarks involving complex reasoning and deep search tasks.

Abstract

Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.

Problem

• exposure bias
• reasoning route convergence
• optimization direction
• self-distillation
• multi-turn agentic settings

Innovation

• Contrastive Learning
• Reinforcement Learning
• Self-Distillation
• Exposure Bias
• Large Language Models