train-test splitting

Machine learning (ML) suffers from weak data curation practices and insufficient documentation of ethical, environmental, and data management information. Method: We systematically evaluated 60 datasets from the NeurIPS Datasets and Benchmarks Track (2021–2023), introducing bibliometric data cataloging theory from library and information science to ML for the first time. We developed a literature-driven, four-dimensional evaluation framework—assessing documentation completeness, ethical impact, environmental footprint, and data management—and designed an actionable, structured rubric alongside an open-source assessment toolkit. Contribution/Results: We released the first exemplar metadata repository showcasing best practices. Our analysis revealed widespread deficiencies across all four dimensions. Based on these findings, we formulated actionable guidelines for conference reviewers and community adoption. All artifacts—including framework, rubric, toolkit, and metadata—are openly shared to advance ML datasets toward higher quality, reusability, and standardization.

Foundation model performance is highly sensitive to training data quality, yet existing data filtering methods rely heavily on hand-crafted rules or heuristic strategies, suffering from poor scalability and computational inefficiency. Method: We propose a learnable data valuation paradigm, introducing for the first time a meta-gradient-driven fine-grained scoring mechanism. By leveraging meta-learning, we model each sample’s marginal contribution to model generalization, and perform end-to-end data value estimation via held-out validation objectives. Contribution/Results: Our approach eliminates reliance on manual rule design, offering inherent adaptability and scalability. Extensive experiments across multiple model scales and datasets demonstrate that, at equivalent model performance, it reduces training FLOPs by up to 40%, significantly improving training computational efficiency.

Manual verification of large-scale dynamic datasets is infeasible, leading to challenges in ensuring data accuracy. Method: This paper proposes a theory-driven, iterative data cleaning framework integrating error detection, automated correction, and progressive optimization. It formally models the iterative process, conducts accuracy testing, and performs probabilistic convergence analysis—validated via simulations and real-world case studies. Contribution/Results: The framework establishes the first rigorous theoretical guarantee of *probabilistic convergence to zero errors* for iterative data cleaning and proves that error detection accelerates error decay. Empirical results demonstrate that it significantly outperforms baseline methods in accuracy improvement and progressively approaches a fully correct dataset state, thereby unifying theoretical soundness with practical efficacy.

Existing certified training algorithms suffer from inconsistent evaluation protocols and suboptimal hyperparameter tuning, leading to incomparable performance claims and unreliable SOTA conclusions. Method: We introduce CTBENCH—the first unified benchmark for certified training—enabling fair, cross-algorithm evaluation of mainstream methods (e.g., IBP, CROWN-IBP, DeepPoly) under a standardized training pipeline, consistent ℓ∞/ℓ2 certification framework, and systematic hyperparameter optimization (grid search + Bayesian optimization). Contribution/Results: Our evaluation reveals that most recently proposed algorithms are substantially overestimated in prior work; after baseline enhancement, their relative improvements drop by over 40% on average. Crucially, all methods achieve significantly higher certified accuracy on CTBENCH than reported in their original papers. This work establishes a reproducible, extensible standard for evaluating certified training, redefining both the robustness training baseline and the SOTA landscape.

Dataset quality defects—such as missing documentation, incorrect labels, and ethical risks—are pervasive in open platforms yet resistant to detection by rule-based scripts, necessitating intelligent, automated identification methods. Method: We introduce the first LLM-agent benchmark for discovering real-world dataset quality issues, covering 221 empirically validated cases across eight platforms. It uniquely evaluates agents’ ability to autonomously detect latent defects without prior prompting. We propose an automated evaluation framework powered by GPT-4o, achieving high agreement with human experts (Cohen’s κ = 0.89), and ensure benchmark reliability via multi-source real-data sampling and expert annotation. Contribution/Results: Experiments reveal that even the state-of-the-art Curator agent detects only ~30% of defects, underscoring task difficulty. All benchmark data, code, and evaluation tools are publicly released to advance intelligent data governance.

Community-driven scientific workflow ecosystems often struggle to sustain themselves due to ambiguous maintenance and user support mechanisms, particularly in cross-platform collaboration and heterogeneous execution environments. This study presents the first cross-platform empirical analysis of the nf-core ecosystem, systematically examining 15,760 GitHub issues, 35,411 pull requests, and 895 forum discussions. By integrating metadata and textual features into predictive models, the research uncovers significant disparities in maintenance and support activities across platforms and highlights weak explicit linkages among them. The findings reveal that issues, pull requests, and forum posts predominantly serve distinct roles—coordinating maintenance, facilitating code integration, and providing user support, respectively. Moreover, issue actionability, diagnostic evidence, and depth of interaction emerge as critical determinants of resolution efficiency.

Existing software modeling datasets are often ad hoc constructions lacking rigorous quality assurance, leading to research findings that are difficult to reproduce, compare, and prone to bias. This work proposes the first benchmarking framework specifically designed for model-driven engineering, treating datasets themselves as first-class evaluation targets. By defining clear metrics for quality, representativeness, and task suitability, the framework establishes a unified platform that enables automated analysis of modeling datasets across multiple languages and formats. For the first time, this approach facilitates systematic evaluation of modeling datasets, substantially enhancing the reproducibility, fairness, and scientific rigor of research in the field.

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

Existing demonstration data filtering metrics often achieve high performance on defect detection benchmarks such as AUROC but do not necessarily improve downstream behavior cloning policy performance. This work introduces controllable structural defects into the LIBERO pick-and-place benchmark to systematically evaluate the impact of various filtering methods on policy training. Experiments reveal a significant disconnect between AUROC and actual policy performance, with most metrics confounded by factors such as episode length. The study advocates evaluating filtering approaches based on policy success rate rather than detection accuracy, demonstrating that an optimal filtering strategy achieves a 90.0% success rate—approaching the 93.3% attained by an oracle using clean data—whereas methods yielding high AUROC scores result in the worst policy performance (13.3%). The authors open-source the testbed and evaluation pipeline.

Current large language model benchmark datasets are often contaminated by inclusion in pretraining corpora, compromising their ability to faithfully assess model generalization. This work systematically defines, for the first time, the core properties of “contamination-resistant” benchmarks and leverages the asymmetry between training and inference in Transformer architectures to propose a cross-model-compatible mathematical formalism that renders data unlearnable during training yet effectively usable during inference. Through contamination detection and theoretical analysis, the study reveals the widespread prevalence of benchmark contamination and establishes a principled framework for designing contamination-resistant benchmarks. The proposed paradigm offers a more reliable foundation for model evaluation and calls upon the research community to adopt such benchmarks to enhance the credibility of performance assessments.