Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

2026-08-07

AI Summary

This work addresses the issue of inflated evaluation scores caused by leakage of public benchmark data into pretraining corpora, which compromises the validity of model assessment. Existing contamination mitigation methods are shown to be significantly overestimated due to flaws in current evaluation metrics. To remedy this, the authors propose SA-PPG, a novel evaluation framework that estimates the per-problem solution probability via Monte Carlo sampling and hierarchically aggregates the resulting discrepancies to more accurately quantify debiasing efficacy. Additionally, they introduce RailCap, a method that dynamically detects contamination traces during autoregressive generation and applies token-level probability capping without requiring prior knowledge of contaminated locations. Experiments demonstrate that SA-PPG exposes the overestimation of existing approaches, while RailCap achieves the lowest SA-PPG scores across multiple contaminated models and benchmarks, confirming its superior ability to recover genuine model performance.

Abstract

Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. \textbf{Contamination mitigation evaluation} intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the \textbf{G-AP} (\textbf{G}ap of \textbf{A}ggregate \textbf{P}erformance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose \textbf{SA-PPG} (\textbf{S}tratified \textbf{A}ggregate of \textbf{P}er-question \textbf{P}robability \textbf{G}aps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. \textbf{RailCap} instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.

Problem

• benchmark contamination
• evaluation metric
• memorization
• model restoration
• performance gap

Innovation

• SA-PPG
• RailCap
• benchmark contamination
• probability gap
• mitigation evaluation