cross-validation design

Existing synthetic data evaluation lacks unified, transferable quantitative metrics. This paper proposes a novel evaluation framework grounded in generalized cross-validation (GCV) and domain transfer learning. It constructs a cross-dataset performance matrix and defines two core metrics: *fidelity*, quantifying distributional similarity between synthetic and real data; and *generalization coverage*, measuring the task-transfer capability of synthetic data across diverse real-world source domains. The framework is model-agnostic and enables normalized, comparative evaluation of detectors such as YOLOv5s across heterogeneous datasets—including Virtual KITTI, KITTI, and BDD100K. Experiments demonstrate that the method effectively quantifies synthetic data quality, significantly enhancing evaluation generality, comparability, and utility for model optimization. It establishes a scalable, reproducible, and standardized evaluation paradigm for synthetic data development.

Traditional model regularization relies on manual hyperparameter tuning to balance model complexity and overfitting. This paper proposes Cross-Reg, a novel cross-validation-inspired regularization framework that for the first time directly leverages validation-set gradients to dynamically optimize regularization strength: training data drive feature learning, while validation data adaptively govern model complexity—eliminating manual intervention entirely. Built upon gradient-based updates with controlled noise injection, Cross-Reg converges efficiently to a near-optimal solution approximating full cross-validation, all within a single training run. It automatically discovers architecture- and task-adapted regularization patterns and natively supports extensions such as data augmentation and uncertainty calibration. Experiments demonstrate substantial improvements in generalization performance and strong robustness to label noise, establishing a new data-driven paradigm for adaptive regularization.

High-complexity machine learning models lack reliable, theoretically grounded mechanisms for detecting overfitting. Method: We propose a statistical hypothesis test that operates solely on training data, dispensing with the need for an independent validation set or PAC-style uniform convergence assumptions. Our approach formalizes overfitting via empirical mean consistency and constructs a rigorous testing framework based on Hoeffding-type concentration inequalities. Contribution/Results: This is the first method to use empirical mean consistency as an overfitting criterion, enabling significance-based inference and implicit diagnosis sensitive to distributional shifts. We prove its validity under mild regularity conditions. Empirical evaluation demonstrates robust identification of overfitting transition points and latent distribution drift, substantially improving both the reliability and interpretability of model selection.

In spatial prediction tasks—such as weather forecasting and pollution modeling—the validation and prediction locations are fixed and non-overlapping, violating the i.i.d. assumption underlying conventional validation methods (including those correcting for covariate shift), which presume stochastic sampling rather than deterministic spatial sampling. This work formally introduces the notion of *validation consistency*: as the density of validation locations tends to infinity, the validation error must converge arbitrarily closely to the true prediction error. Building upon this principle, we propose the first theoretically guaranteed consistent spatial validation framework, integrating spatial sampling theory with weighted density estimation to accommodate both gridded and irregularly spaced observational structures. We prove its consistency under mild regularity conditions. Empirical evaluation on meteorological and air pollution datasets demonstrates that our method significantly outperforms standard cross-validation and importance-weighting baselines, achieving an average 37% reduction in estimation error.

This paper addresses the challenge of quantifying uncertainty in cross-validation (CV) performance estimates—particularly the difficulty of distinguishing true performance differences from random fluctuations during model comparison. We propose an efficient and robust bootstrap-based method that decomposes the variance of CV estimates using a random-effects model, enabling valid statistical inference on CV performance differences without strong modeling assumptions. Compared to standard bootstrap, our approach substantially reduces computational cost while yielding confidence intervals with accurate coverage probability and high statistical power. Extensive evaluations—including simulations and real-world applications across classification, regression, and time-series tasks—demonstrate its strong robustness and generalizability under small-sample settings, non-independent CV folds, and heterogeneous data distributions. The method provides reliable uncertainty quantification to support principled model selection.

Conventional K-fold cross-validation relies on heuristic choices of K (e.g., 5 or 10), leading to suboptimal bias–variance trade-offs in model evaluation. Method: We propose a data- and model-adaptive framework for selecting the optimal K. First, we derive a theoretical upper bound on the estimation uncertainty of cross-validation under finite samples. Then, we formulate a utility-driven optimization objective that explicitly models K-selection as a bias–variance trade-off. Contribution/Results: Empirical validation on real-world datasets—using linear regression and random forests—demonstrates that the optimal K strongly depends on sample size, signal-to-noise ratio, and model complexity; fixed-K conventions thus rest on unwarranted assumptions. Our framework enhances the reliability and interpretability of model evaluation and provides a principled foundation for robust model comparison.

This study addresses the unreliability of performance evaluation in machine learning benchmarking, which often stems from limited test samples and algorithmic stochasticity, thereby hindering accurate assessment of genuine progress. The authors systematically analyze the variance-reduction effect of k-fold cross-validation and introduce a “sample gain” metric to quantify its equivalent data augmentation benefit. Notably, they find that the marginal gains from additional folds decay significantly later than commonly assumed. Leveraging this insight, they propose a dynamic early-stopping strategy that predicts diminishing returns based on initial folds, substantially reducing computational overhead. Experiments on both real-world (e.g., histopathology imaging and NLP fine-tuning) and synthetic datasets demonstrate that the approach markedly enhances the stability and reliability of performance estimation, offering a practical solution for efficient and trustworthy benchmarking.

This study addresses the limitations of conventional cross-validation methods in accurately capturing the complex spatial relationships between training data and target prediction regions, which often leads to biased model performance estimates. To bridge the methodological gap between random and spatial cross-validation, the authors propose a novel paradigm termed “prediction-domain adaptive evaluation.” This framework dynamically tailors the cross-validation strategy to align with the actual prediction scenario by integrating spatial statistics with machine learning evaluation techniques, thereby enabling an adaptive validation workflow. Extensive simulations demonstrate the robustness of the approach across a continuum from interpolation to extrapolation settings. Empirical results show that the proposed method consistently yields more reliable and accurate estimates of predictive accuracy under diverse data distributions.

Scientific machine learning experiments often suffer from distorted performance evaluations due to poor experimental design and inconsistent documentation. To address this, we propose a principled framework for ML experimentation tailored to scientific research, encompassing data preprocessing, model selection, cross-validation, and reporting—emphasizing reproducibility, fair comparison, and transparency. Our key contributions include two novel quantitative metrics: the Logarithmic Overfitting Ratio (LOR) and Composite Overfitting Score (COS), which jointly characterize overfitting severity and instability across cross-validation folds. Complementing these, we introduce standardized preprocessing protocols, rigorously defined strong baselines, and modular visualization templates for diagnostic analysis. Empirical evaluation demonstrates that our framework substantially enhances experimental rigor, reproducibility, and result credibility in scientific ML. It further enables robust performance assessment and cross-study comparability, providing systematic support for establishing reliable benchmarks.

This study investigates the impact of model selection criteria—such as accuracy versus loss—on test performance in neural classifier training, particularly under early stopping with patience. Through systematic empirical evaluation using k-fold cross-validation on standard benchmarks, the work compares multiple validation metrics, including accuracy, cross-entropy, C-Loss, and PolyLoss, under both early stopping and post-hoc full-trajectory selection strategies. The findings reveal that validation loss–based criteria consistently outperform validation accuracy, which exhibits not only inferior performance but also lower stability. More critically, regardless of the selection criterion employed, the chosen models are typically substantially worse than the best test performance observed during training, thereby exposing a fundamental limitation in current model selection paradigms.