design evaluation metrics

This study identifies systematic inconsistencies in the implementation of machine learning evaluation metrics across mainstream programming languages—Python, R, and MATLAB—spanning ten task categories: classification, regression, clustering, statistical testing, image segmentation, and image-to-image translation. Through the first large-scale, cross-platform empirical analysis, we quantitatively assess consistency across 100+ metrics. Results reveal that 36 metrics—including Accuracy, AUC, and MAE—are robust across implementations, whereas critical metrics such as Precision, F1-score, IoU, and Within-Cluster Sum of Squares (WCSS) exhibit substantial discrepancies. To address this, we propose the first comprehensive, task-agnostic standardization roadmap for ML evaluation, accompanied by a curated recommendation list. This work provides both theoretical foundations and practical guidelines to enhance cross-platform reproducibility and result reliability in ML research and deployment.

Existing ML evaluation metric libraries suffer from fragmentation, inconsistent implementations, and weak data validation, hindering reliable cross-framework comparisons. To address this, we propose AllMetrics—the first task-aware, standardized metric framework supporting diverse tasks including regression, classification, clustering, segmentation, and image-to-image translation. AllMetrics systematically eliminates implementation discrepancies (IDs) and reporting discrepancies (RDs) via a modular API, strongly typed input validation, multi-task adapters, and cross-language (Python/Matlab/R) consistency verification. Empirical evaluation across healthcare, finance, and real estate domains demonstrates that AllMetrics significantly reduces evaluation errors, improves reproducibility, and enhances trustworthiness in ML workflows.

In biomedical image segmentation validation, metrics such as the Hausdorff distance suffer from implementation inconsistencies across open-source toolkits, compromising benchmark reliability, introducing biomarker bias, and posing clinical deployment risks. To address this, we systematically evaluate 11 widely used toolkits and introduce, for the first time, a reference implementation based on high-fidelity 3D surface meshes. Our framework integrates real-world clinical data and a cross-platform consistency analysis. Statistical analysis reveals significant inter-tool variation in Hausdorff distance computations (p < 0.001), with interpolation strategy, boundary handling, and sampling density identified as primary sources of discrepancy. Based on these findings, we propose a reproducible and verifiable paradigm for distance-based evaluation, accompanied by standardized computational guidelines. This work substantially enhances the reliability, comparability, and clinical translatability of segmentation assessment.

Machine Learning–Enabled Systems (MLES) suffer from poorly quantifiable and unmanageable complexity, hindering systematic governance. Method: This paper proposes the first measurement-driven architectural model for MLES, innovatively extending the ML system reference architecture to natively support automated collection, cross-dimensional correlation analysis, and visualization of multi-faceted complexity indicators—including training data drift, model iteration coupling, and deployment heterogeneity. The approach integrates architectural modeling, software metrics engineering, and complexity quantification theory to shift complexity assessment from qualitative description to quantitative, traceable decision-making. Contribution/Results: Empirical evaluation demonstrates that the model significantly enhances complexity awareness and decision traceability during architectural evolution. It provides both theoretical foundations and practical tooling for sustainable MLES governance, enabling rigorous, evidence-based architectural management throughout the ML lifecycle.

To address the misalignment between automatic evaluation metrics and human preferences in generative tasks, this paper proposes MetaMetrics—a calibratable meta-metric that supervisely weights and fuses existing metrics to model fine-grained human preferences across multimodal (language/vision), multilingual, and multi-domain settings. Methodologically, it introduces the first preference-dimension-aware metric calibration framework, enabling cross-modal unified evaluation and plug-and-play integration. The approach combines supervised meta-learning, multi-task joint optimization, and explicit modeling of human preference annotations. Experiments demonstrate that MetaMetrics significantly improves correlation with human judgments across multilingual text and vision generation tasks (average Kendall’s τ increase of +18.7%). Moreover, it exhibits strong generalization to unseen domains and models, maintaining robust alignment with human preferences without task-specific retraining.

This work addresses the lack of a universal, tunable, and multi-scenario-compatible metric for data quality assessment, which hinders effective comparison of diverse data cleaning pipelines. To overcome this limitation, the authors propose TOMME—a general-purpose data quality measurement framework based on weighted errors—that extends traditional accuracy into a configurable, composite score. By producing a single quantitative metric, TOMME enables flexible adjustment of error weights according to specific use cases, thereby supporting both automated processing and optimization requirements. Experimental results demonstrate that TOMME exhibits strong adaptability, practicality, and comparability across a variety of scenarios, offering an efficient and unified solution for data quality evaluation and decision-making.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

Real-world distance data are often distorted by noise, missing entries, or violations of the triangle inequality, degrading downstream task performance. This work systematically evaluates the effectiveness of metric repair algorithms, explicitly disentangling two critical subproblems: which edges to repair and how to assign their weights. Through large-scale experiments on both real-world and synthetic non-metric graphs, the study empirically demonstrates—for the first time—that repair efficacy depends primarily on the type and proportion of errors, rather than on error magnitude or graph size. Moreover, accurately identifying the set of edges requiring correction is as crucial as appropriately setting their weights. The findings reveal that most existing repair methods fail to recover a true metric and can even degrade performance, thereby challenging prevailing assumptions in the field.