Learning from Saturated Data: Signals Beyond Correctness for LLM Training

2026-05-31

AI Summary

In domains where large language models (LLMs) have nearly saturated performance, traditional binary correctness labels fail to provide effective training signals. This work proposes replacing such coarse labels with fine-grained quality signals, systematically integrating model self-judged pairwise solution comparisons and token-level entropy—representing uncertainty—into supervised fine-tuning (SFT) and its variants for the first time. Applying this approach to Qwen3-1.7B-Base yields an 18.6% performance gain on simple arithmetic tasks over standard SFT, demonstrating significant improvement. However, on complex benchmarks like GSM8K, gains are limited by the fidelity of the quality signals, highlighting the need for better calibration. By moving beyond the binary correct/incorrect paradigm, this study opens a new pathway for continued optimization in saturated data regimes.

Abstract

The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motivated by this, we investigate whether questions solved with perfect empirical accuracy can nevertheless be used to improve downstream performance. To do so, we replace binary correctness with two sources of more fine-grained quality signals: (1) pairwise LLM self-judgments, in which the model evaluates the relative quality of its own solutions, and (2) token-level entropy, where token-level uncertainty is used as a proxy for solution quality. We incorporate these signals into several training algorithms and evaluate them on Qwen3-1.7B-Base. When training exclusively on a simple arithmetic task, quality-based signals improve performance by up to $18.6\%$ over the base model, substantially outperforming SFT. On GSM8K, however, gains are more modest and depend strongly on the quality signal. For instance, self-judgments show poor agreement with a stronger external judge and can even degrade performance below the base model. Overall, our results suggest that quality-based training can extract useful signal from saturated questions for base models, but that applying such signals to more complex tasks requires careful calibration and further study.

Problem

• saturated data
• large language models
• training signal
• quality signals
• downstream performance

Innovation

• quality signals
• saturated data
• self-judgment
• token-level entropy
• LLM training