🤖 AI Summary
This study addresses a critical yet overlooked issue in the evaluation of AI-driven keyphrase extraction models: severe performance overestimation due to data leakage between training and test sets caused by duplicated samples in benchmark datasets. For the first time, this work systematically quantifies and reveals the misleading impact of such leakage on model generalization assessments. Through large-scale duplicate detection, cross-set similarity analysis, and controlled experiments—augmented by code fingerprinting and semantic comparison to identify leaked instances—the authors propose a rigorous de-duplicated evaluation protocol. Re-evaluation under this protocol demonstrates that state-of-the-art models suffer substantial F1 score drops (exceeding 30% in some cases) once data leakage is eliminated, exposing significant risks in real-world deployment and establishing a more reliable, unbiased benchmark for future research in keyphrase extraction.
📝 Abstract
Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness.