mean absolute error

Standard error metrics (e.g., MAE, MSE) fail to capture dynamical consistency in machine learning predictions—a critical limitation for physical forecasting. This work addresses the frequent distortion of intrinsic dynamical properties—such as chaotic structure and persistence—in long-horizon or recursive predictions. To this end, we propose a novel paradigm for evaluating physical fidelity by introducing two interpretable dynamical fidelity indicators: instantaneous dimension $d$ and inverse persistence $ heta$. We formulate a new error measure grounded in these quantities, revealing an intrinsic relationship between prediction error and system complexity/non-persistence. Our method integrates robust $d$ estimation with $ heta$ computation and is compatible with both direct and recursive prediction frameworks. Extensive validation is conducted on canonical systems—including the Lorenz system, Kuramoto–Sivashinsky equation, Kolmogorov flow—as well as real-world weather data. Results demonstrate that the proposed metrics provide essential diagnostic insights beyond standard errors, enabling principled assessment and improvement of dynamical behavior in learned models.

This work addresses the frequent misalignment between standard mean cross-entropy (Mean CE) and actual task performance in language model training, particularly during fine-tuning and knowledge distillation. By analyzing the dynamics of per-token cross-entropy distributions, the study proposes median cross-entropy (Median CE) and quantile-based summaries as more accurate proxies for model quality. Empirical evaluations on Qwen2.5-1.5B and TinyStories demonstrate that Median CE exhibits significantly stronger agreement with both human and large language model assessments, outperforming Mean CE in tasks involving factual recall and story generation. This research is the first to systematically reveal the decoupling between Mean CE and true model capabilities and establishes Median CE as a more reliable metric for training monitoring, offering a novel perspective for model evaluation and knowledge distillation.

This work addresses the lack of a universal, tunable, and multi-scenario-compatible metric for data quality assessment, which hinders effective comparison of diverse data cleaning pipelines. To overcome this limitation, the authors propose TOMME—a general-purpose data quality measurement framework based on weighted errors—that extends traditional accuracy into a configurable, composite score. By producing a single quantitative metric, TOMME enables flexible adjustment of error weights according to specific use cases, thereby supporting both automated processing and optimization requirements. Experimental results demonstrate that TOMME exhibits strong adaptability, practicality, and comparability across a variety of scenarios, offering an efficient and unified solution for data quality evaluation and decision-making.

Traditional model evaluation relies on single-point metrics, failing to characterize performance stability and uncertainty. This paper proposes a small-sample (10–25 runs) uncertainty quantification framework tailored for high-reliability scenarios. It constructs empirical distributions of performance metrics via repeated stochastic experiments—encompassing random data splits, parameter initializations, and hyperparameter perturbations—and robustly estimates confidence intervals for metric quantiles using bias-corrected nonparametric bootstrap combined with quantile regression. To our knowledge, this is the first systematic approach enabling reliable confidence interval estimation for diverse metrics—including accuracy, F1-score, and MAE—in both classification and regression tasks under small-sample regimes. The method achieves high coverage (>90%) while maintaining narrow interval widths, thereby significantly improving robustness in model selection and enhancing decision-making credibility across multiple benchmark datasets.

Aggregated metrics in machine learning benchmarks—such as multi-task average accuracy—lack statistical reliability due to unquantified uncertainty across tasks. Method: This paper introduces a statistical uncertainty quantification framework comprising three novel components: (i) task-weighted standard error visualization, (ii) Bayesian hierarchical modeling to capture inter-task heterogeneity via shared and task-specific priors, and (iii) task-level bootstrap resampling to explicitly model performance variability and weight uncertainty. Contribution/Results: Evaluated on VTAB, the framework reveals models that rank low globally yet significantly outperform others on specific tasks. It endows aggregated metrics with interpretable confidence intervals, enables principled cross-model comparison, and supports diagnostic bias analysis—thereby mitigating over-simplified interpretations of model performance and enhancing benchmark interpretability and robustness.

Existing evaluation metrics exhibit inconsistent behavior in multimodal machine unlearning tasks, making it difficult to reliably assess unlearning efficacy. This work systematically analyzes the conflicting rankings produced by five widely used metrics across three visual question answering (VQA) benchmarks and proposes a Unified Quality Score (UQS) that achieves more stable performance ranking by weighting each metric according to its distance correlation with an idealized reference model. Empirical evaluation on 36 variants of LLaVA-1.5-7B and BLIP-2 models reveals substantial discrepancies in metric-induced rankings. The proposed UQS demonstrates high stability under 100 random perturbations, achieving a Kendall’s τ of 0.647 ± 0.262. The authors publicly release the benchmark suite, model checkpoints, and an interactive leaderboard to support reproducible research in multimodal unlearning.

This study investigates whether fidelity metrics commonly used in large language model quantization—such as per-token KL divergence—reliably predict downstream task performance. Through a systematic analysis of KL divergence and its variants (including perplexity and Top-1 consistency) against downstream benchmarks, including LiveCodeBench, the authors find that while KL divergence exhibits a strong overall negative correlation with performance (ρ = –0.72 to –0.86), it fails within a “silent zone” near baseline performance levels. Crucially, they demonstrate for the first time that KL divergence primarily captures the magnitude of distributional shift rather than its directionally relevant impact on task outcomes. Consequently, it proves ineffective both as a failure predictor and as a cross-model router, achieving only 42.3%–49.4% accuracy, thereby challenging prevailing assumptions in quantization evaluation.

Current performance evaluation metrics—such as accuracy and F1 score—are typically reported as point estimates, ignoring the uncertainty induced by data clustering structures. This oversight often leads to underestimation of variability and potentially misleading model comparisons. To address this, this work proposes a unified framework that expresses a broad class of performance metrics as smooth functionals of the confusion matrix probabilities. By integrating a cluster-robust sandwich variance estimator, the framework enables valid confidence interval construction, hypothesis testing, and paired model comparison. It represents the first systematic application of cluster-robust inference to predictive performance evaluation, accommodating both binary and multiclass settings, and further provides asymptotic theory–based methods for power and sample size calculations. Simulations demonstrate that the proposed approach achieves near-nominal coverage across diverse dependence structures and substantially outperforms conventional methods that ignore clustering; real-data analyses confirm that accounting for clustering can materially alter evaluation conclusions.

This study addresses the inconsistency of existing prediction evaluation metrics—such as ABC and Gini—with the principle of mean consistency, stemming from their reliance on predicted values for weighting, which can lead to erroneous model selection. Building upon Bregman divergences, the authors develop a mean-consistent loss framework, rederive the Murphy decomposition to disentangle prediction error into calibration and discrimination components, and establish a theoretical link between these components and Lorenz-curve-based metrics. They propose a new metric, ABC², to enhance sensitivity to mean calibration, and demonstrate that ABC, ABC², and Gini all violate mean consistency due to prediction-dependent weighting. Furthermore, they prove the equivalence between the number of crossings in Lorenz and Murphy curves and, under a single-crossing condition, provide a weak dominance criterion for predictive superiority, offering both theoretical grounding and practical guidelines for reliable model evaluation.