Learning from the Gap Between Pass@K and Pass@1

2026-09-17

AI Summary

This study addresses the limited single-pass inference capability and high search costs of large language models, as well as the budget inefficiency caused by uniform sampling in post-training. We propose GapFT, a method that targets the performance gap between Pass@K and Pass@1. By leveraging correctness discrepancies for data filtering, GapFT applies rejection-sampling fine-tuning exclusively to instances that fail in a single pass but become solvable under multiple sampling attempts. This efficiently distills search-derived gains into model weights while decomposing the optimization objective into failure correction and regression prevention. Experiments demonstrate that GapFT consistently outperforms uniform rejection fine-tuning on benchmarks such as LogiQA. Notably, it recovers most of the performance achieved by full-data fine-tuning using only approximately one-third of the samples, confirming the high training utility of shallow search failure points.

Abstract

Large language models (LLMs) are increasingly trained with reinforcement learning from verifiable rewards (RLVR). An exact verifier can also support test-time scaling by selecting a passing response from multiple samples, while other deployments use beam search, adaptive sampling, or tools. We study single-sample decoding, where each query receives one response without search, to ask whether search-exposed behavior can be absorbed into the model. Existing verified-response post-training recipes do not generally distinguish problems already solved on the first decode from failures recovered within K samples. Under a fixed budget, this can spend examples repeating behavior the deployed policy already has. We introduce GapFT, which selects training evidence by the source checkpoint's single-sample outcome and fine-tunes on the Pass@K-Pass@1 gap: problems the policy fails on one sample but solves within K samples. We match training examples, processed tokens, and optimizer steps while keeping the objective unchanged. GapFT fills the matched budget with recovered failures and uses an exact decomposition to distinguish corrections of recovered and missed failures from regressions on first-decode successes. On LogiQA 2.0 and ReClor with Llama-3.1-8B, GapFT improves Pass@1 by 14.4 and 13.9 points over the source model, outperforms budget-matched uniform verified RFT at the same learning rate, and matches fine-tuning on the full verified pool using one third of the data. A single decode matches the source model's verifier-selected Pass@4 accuracy. A randomized control attributes gains to covering distinct failures, and our analysis relates available gains to transferable failure support. A three-seed Qwen2.5-7B replication retains positive gains over uniform RFT on both logic tasks.

Problem

• Pass@K
• Pass@1
• rejection sampling fine-tuning
• training budget allocation
• large language models

Innovation

• GapFT
• Rejection Sampling Fine-Tuning
• Pass@K-Pass@1 Gap
• Verifier-guided Training
• Budget Allocation