RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

2026-10-08

AI Summary

This study addresses the overconfidence and performance degradation under distribution shift exhibited by large reasoning models during reinforcement learning due to inadequate calibration. To this end, we propose RL-ARC, a framework that innovatively incorporates reasoning confidence as an auxiliary signal alongside answer confidence to apply guided regularization on correct samples and overconfidence penalties on incorrect ones. By integrating verifiable reward-based reinforcement learning with uncertainty estimation techniques, RL-ARC significantly improves model calibration in both in-distribution (ID) and out-of-distribution (OOD) scenarios. The proposed approach endows models with adaptive confidence estimation capabilities without substantially compromising their reasoning performance.

Abstract

Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.

Problem

• calibration
• overconfidence
• reasoning models
• reinforcement learning
• distribution shift

Innovation

• Calibration
• Reasoning Confidence
• Uncertainty Estimation
• Reinforcement Learning
• Distribution Shift