rmse computation

This work addresses the challenge that existing model evaluation methods often fail to reliably assess estimator quality in low-variance settings due to confounding between bias and variance or excessive sensitivity of statistical tests. To overcome this limitation, the authors propose a fault-tolerant evaluation framework that unifies bias and variance modeling through an adjustable tolerance parameter ε, enabling robust assessment of sample-efficient performance estimators within practically acceptable error margins. The framework integrates bias-variance analysis, fault-tolerant evaluation theory, and an adaptive ε-optimization algorithm, making it particularly well-suited for scenarios with low annotation costs. Experimental results demonstrate that the proposed approach provides a more comprehensive and reliable characterization of estimator behavior, significantly enhancing both the practical utility and stability of performance evaluation.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

Existing nonlinear model selection predominantly relies on performance metrics while neglecting sampling variability and lacking rigorous statistical tests for homoscedasticity of prediction errors. To address this, we propose a robustified Morgan–Pitman test for variance equality, incorporating residual decorrelation preprocessing and heavy-tailed distribution–adaptive robust estimation. This approach markedly enhances robustness against high-variance outliers and non-normal errors without assuming any specific error distribution, making it broadly applicable to prediction error comparisons across arbitrary machine learning models. Simulation studies and real-data experiments demonstrate that the method maintains high statistical power and stringent type-I error control—even under small-sample, heteroscedastic, and heavy-tailed conditions. Consequently, it provides an interpretable, reproducible, and distribution-free statistical criterion for principled model selection.

This work addresses the lack of a universal, tunable, and multi-scenario-compatible metric for data quality assessment, which hinders effective comparison of diverse data cleaning pipelines. To overcome this limitation, the authors propose TOMME—a general-purpose data quality measurement framework based on weighted errors—that extends traditional accuracy into a configurable, composite score. By producing a single quantitative metric, TOMME enables flexible adjustment of error weights according to specific use cases, thereby supporting both automated processing and optimization requirements. Experimental results demonstrate that TOMME exhibits strong adaptability, practicality, and comparability across a variety of scenarios, offering an efficient and unified solution for data quality evaluation and decision-making.

This study systematically evaluates the impact of model quantization on the correctness and resource efficiency of deep learning systems, while also exploring methodologies for cross-study evidence aggregation in data-driven empirical research. Methodologically, it innovatively applies Structured Synthesis Methods (SSM) for the first time in this domain, integrating findings from six empirical studies covering 19 models through a qualitative-quantitative mixed analysis. Results demonstrate that quantization yields substantial resource gains—average storage compression of ×3.2, inference latency reduction of −41%, and GPU energy consumption decrease of −38%—with only a marginal correctness degradation (−1.7% on average), representing a well-controlled trade-off. The study identifies both consistent patterns and fragmentation bottlenecks in quantization effects, and proposes a refined empirical research framework and methodological guidelines tailored to quantization techniques. These contributions provide foundational methodological support and practical guidance for optimizing trustworthy AI systems.

This study addresses the critical issue of model instability in software engineering optimization, which leads to substantial variability across repeated experiments and undermines both credibility and practical utility. Rather than treating instability as mere random noise, this work conceptualizes it as a quantifiable and manageable property that should be integrated into standard evaluation frameworks. By systematically modulating label usage, model complexity, and partition scoring strategies—combined with multi-objective optimization, causal intervention, data locality analysis, and model calibration—the proposed approach significantly enhances result consistency. Empirical evaluation demonstrates that the optimized configuration reduces the standard deviation of error by 22% on average and outperforms default settings in 119 out of 127 datasets, achieving a 4.8-fold improvement in result consistency.

This work addresses the challenge of reliably detecting performance degradation in large language models caused by optimization techniques such as quantization, where observed drops in accuracy may stem from genuine model deterioration or mere evaluation noise. To this end, the authors propose a statistical hypothesis testing framework based on McNemar’s test, which introduces sample-level paired comparisons for the first time in the context of LLM degradation analysis, thereby overcoming the limited sensitivity of conventional task-level aggregation. Integrated with multi-benchmark accuracy aggregation and the LM Evaluation Harness, the method effectively controls false positive rates and reliably identifies performance degradations as small as 0.3%. Empirical results demonstrate that the approach accurately flags models exhibiting true degradation while producing no false alarms for theoretically lossless optimizations.

Fixed-size benchmarking in model evaluation often fails to balance efficiency, statistical reliability, and diverse objectives, leading to either excessive resource consumption or unreliable results. This work proposes the first adaptive framework that integrates sequential testing into AI model evaluation, dynamically allocating evaluation data based on stopping criteria tailored for model ranking and selection tasks. By combining sequential hypothesis testing, minimum detectable effect analysis, and diminishing returns detection, the method achieves substantial gains in efficiency without compromising rigor. Empirical validation on the Open VLM Leaderboard demonstrates an 80% reduction in computational cost while maintaining a confidence interval width of 2.5 points, significantly enhancing both the practicality and scalability of model evaluation.

Existing software modeling datasets are often ad hoc constructions lacking rigorous quality assurance, leading to research findings that are difficult to reproduce, compare, and prone to bias. This work proposes the first benchmarking framework specifically designed for model-driven engineering, treating datasets themselves as first-class evaluation targets. By defining clear metrics for quality, representativeness, and task suitability, the framework establishes a unified platform that enables automated analysis of modeling datasets across multiple languages and formats. For the first time, this approach facilitates systematic evaluation of modeling datasets, substantially enhancing the reproducibility, fairness, and scientific rigor of research in the field.

This work addresses the frequent lack of systematic and credible statistical evaluation in ECE/CS research, which often undermines the persuasiveness of empirical claims. To bridge this gap, we propose a structured statistical evaluation workflow tailored for beginners, integrating classical methods—such as t-tests and ANOVA—with modern nonparametric techniques, including bootstrap resampling, Wilcoxon tests, and Cliff’s delta. The framework spans the entire pipeline from formulating research claims to reporting results, supporting factorial designs, multiple comparison corrections, and simulation-based validation. Accompanying the methodology are fully reproducible Python implementations, illustrative examples, and a pre-submission checklist. This approach substantially enhances the reliability and reproducibility of experimental findings while offering both pedagogical utility and practical guidance for researchers.