Cross-Model Disagreement as a Label-Free Correctness Signal

2026-03-26

AI Summary

This work addresses the challenge of reliably detecting incorrect yet high-confidence responses from large language models in the absence of ground-truth labels. The authors propose leveraging cross-model disagreement as an unsupervised signal of answer correctness by introducing Cross-Model Perplexity (CMP) and Cross-Model Entropy (CME)—metrics derived from the perplexity and entropy of a verification model on the generated answers. Requiring only a single forward pass and no additional training or content generation, this approach uniquely transforms inter-model divergence into a training-free, label-free uncertainty estimator. Evaluated on benchmarks including MMLU, TriviaQA, and GSM8K, the method substantially outperforms intrinsic uncertainty baselines; notably, CMP achieves an AUROC of 0.75 on MMLU, significantly exceeding the baseline of 0.59.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment. Existing approaches rely on a model's own uncertainty -- such as token entropy or confidence scores -- but these signals fail critically on the most dangerous failure mode: confident errors, where a model is wrong but certain. In this work we introduce cross-model disagreement as a correctness indicator -- a simple, training-free signal that can be dropped into existing production systems, routing pipelines, and deployment monitoring infrastructure without modification. Given a model's generated answer, cross-model disagreement computes how surprised or uncertain a second verifier model is when reading that answer via a single forward pass. No generation from the verifying model is required, and no correctness labels are needed. We instantiate this principle as Cross-Model Perplexity (CMP), which measures the verifying model's surprise at the generating model's answer tokens, and Cross-Model Entropy (CME), which measures the verifying model's uncertainty at those positions. Both CMP and CME outperform within-model uncertainty baselines across benchmarks spanning reasoning, retrieval, and mathematical problem solving (MMLU, TriviaQA, and GSM8K). On MMLU, CMP achieves a mean AUROC of 0.75 against a within-model entropy baseline of 0.59. These results establish cross-model disagreement as a practical, training-free approach to label-free correctness estimation, with direct applications in deployment monitoring, model routing, selective prediction, data filtering, and scalable oversight of production language model systems.

Problem

• label-free correctness detection
• confident errors
• cross-model disagreement
• language model reliability
• uncertainty estimation

Innovation

• cross-model disagreement
• label-free correctness
• Cross-Model Perplexity
• confident errors
• training-free uncertainty