Test-time Calibration Learning for Large Language Model Reasoning

2026-10-01

AI Summary

Confidence calibration for large language models (LLMs) typically relies on labeled data, hindering adaptation to novel tasks in label-scarce scenarios. This work proposes Test-Time Calibration Learning (TTCL), a framework that leverages unlabeled data for self-supervised joint optimization. Methodologically, TTCL synergizes multi-response self-supervised signal generation with reinforcement learning to co-optimize inference accuracy and confidence estimation, which is theoretically proven to serve as a bounded proxy for the ideal objective. Experimental results demonstrate that TTCL improves baseline model accuracy by 40.13% and reduces Expected Calibration Error (ECE) by 70.80%, significantly enhancing cross-domain reliability.

Abstract

Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incorporate calibration learning into reinforcement learning (RL), jointly optimizing answer correctness and verbalized confidence using ground-truth correctness supervision. However, their reliance on labeled data limits their applicability in practical test-time settings, where ground-truth labels are unavailable and calibration may need to adapt to newly encountered target tasks. To address this challenge, we propose Test-Time Calibration Learning (TTCL), a label-free framework that jointly adapts reasoning accuracy and verbalized confidence directly on unlabeled target-task data. Specifically, TTCL derives self-supervision signals for both correctness and calibration from multiple model-generated responses, enabling calibration learning at test time without ground-truth labels. Theoretical analysis further establishes TTCL as a bounded surrogate for the ideal calibration objective. Extensive experiments on mathematical reasoning and factual question answering demonstrate that TTCL consistently improves both accuracy and calibration across diverse models and tasks. On base models, TTCL achieves an average relative accuracy improvement of +40.13% and an ECE reduction of +70.80% across eight benchmarks. Moreover, TTCL can further improve both accuracy and calibration for already calibrated models under domain shift, particularly when source-domain calibration transfers poorly to target tasks. In the math-to-factQA setting, TTCL achieves an average relative accuracy gain of +20.35% and reduces ECE by +53.83%. The source code is released at https://github.com/tmlr-group/TTCL.

Problem

• Test-time calibration
• Large language models
• Label-free learning
• Verbalized confidence
• Domain shift

Innovation

• Test-Time Calibration Learning
• Label-Free Framework
• Self-Supervision
• Verbalized Confidence
• Domain Shift