🤖 AI Summary
Existing synthetic data contamination detection methods rely solely on token-level overlap, failing to identify latent benchmark contamination—where no lexical repetition occurs but semantic similarity persists—thereby severely compromising model evaluation validity. To address this, we propose the first four-tiered contamination detection framework, spanning token-level, semantic, reasoning-pattern, and performance-collapse dimensions. Our approach integrates semantic embedding comparison, reasoning-path analysis, anomalous performance monitoring, and controlled experimental design to systematically uncover contamination across semantic and reasoning levels. Extensive experiments on MMLU, GSM8K, and HumanEval demonstrate that our method achieves an average F1-score of 0.76—outperforming the state-of-the-art by 26.5%—significantly enhancing the reliability of synthetic data auditing and the credibility of model evaluations.
📝 Abstract
Synthetic data has become essential for training foundation models, yet benchmark contamination threatens evaluation integrity. Although existing detection methods identify token-level overlap, they fail to detect semantic-level contamination where synthetic data conceptually resemble benchmarks without lexical overlap. This gap is critical as foundation models increasingly train on synthetic data that may implicitly encode benchmark knowledge. We propose a hierarchical contamination detection framework operating at four levels: token level, semantic level, reasoning pattern, and performance cliff detection. Through controlled experiments on MMLU, GSM8K and HumanEval, we demonstrate that semantic-level contamination evades existing methods (F1=0.17-0.49) but is effectively detected by our hierarchical approach (F1 = 0.76), with an average improvement of 26. 5% over state-of-the-art baselines. Our framework provides practitioners with practical tools for audit pipelines and enables responsible deployment of synthetic training data.