CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning

2026-08-03

AI Summary

This work addresses two critical challenges in applying large language models to reinforcement learning: insufficient precision in trajectory-level feedback and problem difficulty drift during training. To tackle these issues, the authors propose a trajectory-level value variance estimation method that quantifies the intrinsic stochasticity of the generation process and leverages this estimate to design a variance-aware advantage function adjustment mechanism. Additionally, they introduce a dynamic curriculum learning strategy that adaptively selects training problems based on the model’s current capability. This combined approach effectively mitigates difficulty drift and enhances exploration efficiency. Empirical results demonstrate significant improvements over strong baselines such as VAPO across multiple mathematical reasoning benchmarks, yielding higher reasoning accuracy and robustness.

Abstract

Reinforcement learning (RL) has emerged as an effective method for enhancing the reasoning capabilities of large language models (LLMs). However, existing methods suffer from insufficient precision in feedback on generated answer trajectories and exhibit the phenomenon of problem difficulty drift. To address these challenges, we propose CVPO - Curriculum-guided Value-Variance Policy Optimization. At the response trajectory level, we find that token-level value-variance correlates with exploration intensity. Our theoretical analysis shows this variance bounds policy update magnitude. We then use the estimated trajectory value-variance to quantify the intrinsic randomness in generation. Based on this, we design a variance-aware advantage adjustment mechanism for different reward types. At the question level, we introduce a dynamic curriculum weighting method that adapts to question difficulty. This helps the model focus on tasks matched to its current ability during each training stage. Experimental results show our method outperforms strong value-based baselines like VAPO. It achieves better performance and stronger exploration, enabling more accurate and robust reasoning in language models across various math tasks.

Problem

• reinforcement learning
• reasoning
• feedback precision
• difficulty drift
• large language models

Innovation

• value-variance adaptation
• dynamic curriculum learning
• reinforcement learning
• large language models
• reasoning