The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning

2026-06-05

AI Summary

Standard GRPO training treats easy, hard, and learnable problems uniformly, leading to inefficient use of computational resources. This work uncovers three key phenomena during training—confidence inflation, advantage collapse, and hierarchical convergence—and introduces CoDaPO, a method that assigns bounded values to problems by integrating reasoning confidence and empirical difficulty. CoDaPO dynamically reweights policy updates and resamples high-value learnable examples. Built upon an analytical framework incorporating token log-probabilities, group-normalized advantages, and token-level weights, the approach synergistically combines with policy gradient optimization. Evaluated across twelve reasoning benchmarks, CoDaPO significantly outperforms existing reinforcement learning methods and consistently improves accuracy under fixed computational budgets.

Abstract

RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.

Problem

• LLM reasoning
• policy optimization
• difficulty adaptation
• compute efficiency
• reinforcement learning

Innovation

• difficulty-adaptive
• confidence-based weighting
• policy optimization
• learnable band
• LLM reasoning