CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

2026-03-18

AI Summary

In label-free reinforcement learning, majority-vote pseudo-labeling often leads to a “consensus trap,” causing collapse in output diversity and systematic errors. To address this issue, this work proposes CoVerRL, a novel framework that introduces, for the first time, a generator–verifier co-evolution mechanism. A single model alternates between generator and verifier roles during training, enabling the verifier to progressively filter out self-consistent errors and establish a virtuous cycle where reasoning and self-verification capabilities mutually reinforce each other. Experimental results demonstrate that CoVerRL outperforms existing label-free methods by 4.7–5.9% on mathematical reasoning benchmarks when applied to Qwen and Llama family models, while boosting self-verification accuracy from approximately 55% to over 85%.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9\% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55\% to over 85\%, confirming that both capabilities genuinely co-evolve.

Problem

• label-free reinforcement learning
• consensus trap
• self-consistency
• systematic errors
• pseudo-labels

Innovation

• CoVerRL
• consensus trap
• generator-verifier co-evolution
• label-free reinforcement learning
• self-verification