Technology Category
Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)Reasoning under Uncertainty: Other Foundations of Reasoning under UncertaintyCognitive Modeling & Cognitive Systems: Analogy
How the Misuse of a Dataset Harmed Semantic Clone Detection
🤖 AI Summary
This paper identifies a systemic bias arising from the misuse of BigCloneBench as a ground-truth benchmark for semantic clone detection: 93% of its Weak Type-3/Type-4 clone pairs (N=406, manually verified) are functionally dissimilar, indicating severe label inaccuracy. Method: We conduct bibliometric analysis across 179 papers and apply a truth-quality assessment framework to evaluate ground-truth reliability. Contribution/Results: Our analysis casts doubt on the validity of conclusions in 139 semantic clone studies; high F1 scores often reflect overfitting to spurious dataset patterns rather than genuine semantic understanding. This work is the first to systematically demonstrate—via large-scale manual adjudication, statistical sampling, and meta-analysis—the substantive threat posed by dataset misuse to research validity in this domain. We rigorously establish that BigCloneBench is suitable only for syntactic or textual clone evaluation; its application beyond this scope risks generating misleading scientific conclusions.
Technology Category
Application Category
📝 Abstract
BigCloneBench is a well-known and widely used large-scale dataset for the evaluation of recall of clone detection tools. It has been beneficial for research on clone detection and has become a standard in evaluating the performance of clone detection tools. More recently, it has also been widely used as a dataset to evaluate machine learning approaches to semantic clone detection or code similarity detection for functional or semantic similarity. This paper demonstrates that BigCloneBench is problematic to use as ground truth for learning or evaluating semantic code similarity, and highlights the aspects of BigCloneBench that affect the ground truth quality. A manual investigation of a statistically significant random sample of 406 Weak Type-3/Type-4 clone pairs revealed that 93% of them do not have a similar functionality and are therefore mislabelled. In a literature review of 179 papers that use BigCloneBench as a dataset, we found 139 papers that used BigCloneBench to evaluate semantic clone detection and where the results are threatened in their validity by the mislabelling. As such, these papers often report high F1 scores (e.g., above 0.9), which indicates overfitting to dataset-specific artefacts rather than genuine semantic similarity detection. We emphasise that using BigCloneBench remains valid for the intended purpose of evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge the important contributions of BigCloneBench to two decades of traditional clone detection research. However, the usage of BigCloneBench beyond the intended purpose without careful consideration of its limitations has led to misleading results and conclusions, and potentially harmed the field of semantic clone detection.
Problem
Research questions and friction points this paper is trying to address.
BigCloneBench mislabels semantic clone pairs, harming evaluation accuracy
Many papers report inflated F1 scores due to dataset mislabeling
Misuse of BigCloneBench misleads semantic clone detection research
Innovation
Methods, ideas, or system contributions that make the work stand out.
Identifies mislabelling in BigCloneBench dataset
Analyzes impact on semantic clone detection validity
Recommends cautious usage for intended purposes only
🔎 Similar Papers
No similar papers found.
