🤖 AI Summary
This study addresses the prevalent yet underexplored issue of refactorable repetitive step subsequences in Behavior-Driven Development (BDD) tests, for which no automated identification and classification methods previously existed. We propose the first end-to-end framework that leverages Sentence-BERT, UMAP, and HDBSCAN to perform semantic clustering on Gherkin corpora, thereby uncovering recurring fragments. These fragments are then annotated manually to train an XGBoost classifier that ranks their refactoring potential and assigns them to one of three established refactoring patterns. Applying our approach across 339 repositories, we identify 692,020 repetitive patterns and release the first large-scale annotated dataset, along with a complete toolchain and evaluation benchmark. Experimental results demonstrate that our XGBoost classifier achieves an F1 score of 0.891, significantly outperforming rule-based baselines and LLM-based judges, with 75% of test scenarios containing high-potential refactoring candidates.
📝 Abstract
Context. Behaviour-Driven Development (BDD) software test suites accumulate duplicated step subsequences. Three published refactoring patterns are available (within-file Background, within-repo reusable-scenario invocation, cross-organisational shared higher-level step), but no prior work automates which recurring subsequences are worth extracting or which mechanism applies. Objective. Rank recurring step subsequences ("slices") by refactoring suitability (extraction-worthy), pre-map each to one of the three patterns, and quantify prevalence across the public BDD ecosystem. Method. Every contiguous L-step window (L in [2, 18]) in a 339-repository / 276-upstream-owner Gherkin corpus is keyed by paraphrase-robust cluster identifiers and counted under three scopes. Sentence-BERT (SBERT) / Uniform Manifold Approximation and Projection (UMAP) / Hierarchical Density-Based Clustering (HDBSCAN) recovers paraphrase-equivalent slices. Three authors label a stratified 200-slice pool against a written rubric. An eXtreme Gradient Boosting (XGBoost) extraction-worthy classifier trained under 5-fold cross-validation is compared with a tuned rule baseline and two open-weight Large Language Model (LLM) judges. Results. The miner produces 5,382,249 slices collapsing to 692,020 recurring patterns. Three-author Fleiss' kappa = 0.56 (extraction-worthy) and 0.79 (mechanism). The classifier reaches out-of-fold F1 = 0.891 (95% CI [0.852, 0.927]), outperforming both the rule baseline (F1 = 0.836, p = 0.017) and the better LLM judge (F1 = 0.728, p < 1e-4). 75.0%, 59.5%, and 11.7% of scenarios carry a within-file Background, within-repo reusable-scenario, or cross-organisational shared-step candidate. Conclusion. Paraphrase-robust subscenario discovery yields a corpus-wide census of BDD refactoring opportunities; pipeline, classifier predictions, labelled pool, and rubric are released under Apache-2.0.