🤖 AI Summary
Standard k-fold cross-validation suffers from sample reuse: each instance participates in training for $k-1$ folds and testing once, inducing training-set overlap, evaluation bias, and inflated variance estimates. To address this, we propose Single-Use k-Fold Cross-Validation (SU-CV), the first method ensuring each sample is used *exactly once* for training and *exactly once* for testing. SU-CV achieves this via non-overlapping, stratified partitioning—eliminating training-set redundancy while preserving class proportions. It is model-agnostic, requires no architectural modifications, and integrates seamlessly with any classifier. Empirical results demonstrate that SU-CV significantly reduces estimator variance (yielding more conservative performance estimates), mitigates overfitting tendencies, and cuts training computational cost by approximately $(k-1)/k$. Extensive evaluation across multiclass benchmarks confirms its stability and generalization robustness. SU-CV establishes a theoretically sound, fairer, and more efficient benchmark for model evaluation.
📝 Abstract
In traditional k-fold cross-validation, each instance is used ($k!-!1$) times for training and once for testing, leading to redundancy that lets many instances disproportionately influence the learning phase. We introduce Irredundant $k$--fold cross-validation, a novel method that guarantees each instance is used exactly once for training and once for testing across the entire validation procedure. This approach ensures a more balanced utilization of the dataset, mitigates overfitting due to instance repetition, and enables sharper distinctions in comparative model analysis. The method preserves stratification and remains model-agnostic, i.e., compatible with any classifier. Experimental results demonstrate that it delivers consistent performance estimates across diverse datasets --comparable to $k$--fold cross-validation-- while providing less optimistic variance estimates because training partitions are non-overlapping, and significantly reducing the overall computational cost.