Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

2026-08-04

AI Summary

This work addresses the instability in test-time reinforcement learning (TTRL) caused by unreliable pseudo-labels generated via majority voting, where fluctuating consensus strength often leads to ineffective policy updates or vanishing gradients. To mitigate this issue, the paper proposes an MCMC-based prompt sampling mechanism that dynamically adjusts consensus strength into a target interval during rollouts. By employing a power-transformed prefix distribution, the method enables approximate finite-step sampling that adaptively guides prompt selection. This approach substantially enhances the stability and efficacy of policy updates, consistently outperforming standard TTRL across multiple datasets and backbone models. Ablation studies further confirm the critical role of the proposed consensus regulation mechanism in achieving these gains.

Abstract

Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.

Problem

• test-time reinforcement learning
• consensus strength
• pseudo-labels
• reward signal
• majority voting

Innovation

• test-time reinforcement learning
• consensus regulation
• hint sampling
• MCMC
• power-transformed distribution