supervised evaluation

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

For high-stakes decision-making domains lacking ground-truth labels—such as judicial adjudication and clinical diagnosis—this paper proposes a human-centered classifier evaluation framework. Methodologically, it introduces the *Rater Equivalence Number* (REN), the first formal metric quantifying how many human raters’ collective judgment a model’s performance equivalently matches, thereby enabling interpretable, human-aligned assessment. The framework distinguishes two utility models: *ground-truth consistency* (agreement with latent consensus) and *individual judgment matching* (fidelity to diverse human judgments), supporting value-sensitive deployment trade-offs. It integrates crowdsourced annotation modeling, statistical consistency analysis, and benchmark panel construction to jointly generate an evaluation reference standard and quantify model performance from human-labeled data. Empirical case studies and formal analysis validate its theoretical soundness and practical efficacy, establishing an actionable evaluation paradigm and deployment guidance for AI systems operating without gold-standard labels.

Conventional evaluation metrics (e.g., accuracy, F1) fail to reflect the practical value of AI models in human-AI collaborative settings—particularly when low-confidence predictions require human intervention—leading to suboptimal model selection. Method: We propose a task-aware value-oriented evaluation paradigm grounded in cost-sensitive decision theory, jointly modeling costs of correct predictions, misclassifications, and human review. Crucially, we emphasize probability calibration—not model complexity—as the primary determinant of real-world value, enhancing reliability via Platt scaling or temperature scaling. Contribution/Results: Empirical validation across diverse operational scenarios demonstrates that standard metrics frequently select inferior models, whereas simple yet well-calibrated models substantially increase end-to-end workflow value. Our framework provides an interpretable, optimization-friendly, value-driven assessment methodology for enterprise-scale AI deployment.

Current machine learning evaluation practices predominantly rely on surface-level performance metrics, often neglecting the internal mechanisms of models. This work proposes trustworthy interpretability as a central evaluation paradigm and, for the first time, systematically demonstrates that it satisfies core criteria from the philosophy of science—namely falsifiability, reproducibility, and predictive power. By constructing an evaluation framework that integrates causal analysis with mechanistic probing, the study delineates three functional pathways through which interpretability enables the identification of behavioral origins, detection of latent flaws, and prediction of potential failure modes. This approach advances model assessment beyond performance-oriented benchmarks toward a deeper understanding of underlying mechanisms.

Existing feature interpretability methods suffer from poor alignment with domain-specific expert knowledge, particularly in high-dimensional data where such knowledge is difficult to formalize mathematically. Method: This paper introduces FIX—the first Feature Interpretability benchmark for eXpert-knowledge alignment—built through interdisciplinary collaboration across cosmology, psychology, and medicine, and across vision, language, and time-series modalities. FIX establishes a structured knowledge encoding framework and a human-in-the-loop evaluation protocol, culminating in a unified quantitative metric: FIXScore. Contribution/Results: FIX enables the first expert-driven, cross-domain, cross-modal assessment of feature group consistency. Evaluated on six real-world tasks, mainstream methods (e.g., Grad-CAM, SHAP) achieve FIXScores consistently below 0.3, revealing severe misalignment with expert judgment. FIX provides a reproducible, comparable, and domain-grounded evaluation paradigm for explainable AI.

This work addresses the trade-off between human annotation cost and system accuracy in human-in-the-loop classification by proposing an optimization framework based on a dual-threshold strategy. By setting upper and lower confidence thresholds, the system automatically processes high-certainty samples while routing only ambiguous cases to human reviewers. The approach formalizes the human-AI collaboration problem, identifies the critical region where human intervention yields diminishing returns, and quantifies the marginal benefit of manual review across diverse scenarios through probabilistic score modeling, Monte Carlo simulation, and optimization algorithms. Empirical evaluations demonstrate the framework’s generality and effectiveness across multiple domains—including entity resolution, fraud detection, medical triage, and content moderation—achieving high accuracy while substantially reducing human workload.

Traditional machine learning treats systematic variations in human annotations—such as ambiguity, interpretive disagreement, and errors—as mere noise, thereby obscuring the true sources of error. This work reframes annotation as a measurement process and introduces the first statistical framework that simultaneously accommodates both shared and individualized notions of “ground truth.” The proposed model decomposes annotation variability into four interpretable components: instance difficulty, annotator bias, contextual noise, and relational consistency. Leveraging probabilistic modeling, we estimate and validate these components on multi-annotator natural language inference data. Empirical results confirm the presence of all four sources of variation and demonstrate that the framework effectively disentangles annotator behavior, offering actionable insights for constructing higher-quality datasets.

This work addresses the lack of reliability, interpretability, and consistency analysis in existing automated evaluation methods for multimodal tasks. To this end, it proposes the first interpretable evaluation framework tailored for large-scale multimodal benchmarks, introducing a dedicated multimodal judge model that integrates textual, audio, visual, and video signals. Rather than relying solely on scalar scores, the framework generates diagnostic feedback through reasoning consistency analysis. Leveraging a multimodal large language model architecture and publicly available datasets with fixed-seed sampling, the method evaluates multiple state-of-the-art models on 280 samples. The results demonstrate strong alignment with human annotations, confirming the framework’s reliability, scalability, and comprehensive capability in assessing the output quality of multimodal foundation models.

This study investigates whether existing algorithmic evaluation metrics for counterfactual explanations align with users’ perceptions of explanation quality. Through user studies conducted on three datasets, the authors systematically compare widely used algorithmic metrics against multidimensional human subjective ratings of counterfactual explanations. Employing correlation analyses and multivariate regression models, they assess the consistency and predictive power of these metrics. The findings reveal that algorithmic metrics generally exhibit weak correlations with human judgments and are highly dataset-dependent. Moreover, increasing the number of metrics yields only marginal improvements in predictive performance. These results expose structural limitations in current evaluation practices, underscoring their inability to capture key aspects of explanation quality that matter to users, and provide empirical support for advancing human-centered evaluation paradigms in explainable AI.

This study addresses the reliability limitations of human evaluation in AI model assessment, which are often compromised by rater biases such as severity and central tendency. To mitigate these issues, the work systematically introduces the Multifaceted Rasch Model—a psychometric framework grounded in Item Response Theory—into the human evaluation pipeline for AI-generated outputs. This approach explicitly models rater effects, enabling effective disentanglement of the true textual quality from systematic scoring biases. Empirical validation on the OpenAI Summarization dataset demonstrates that the proposed method significantly corrects for rater bias, thereby enhancing construct validity and transparency in evaluation. Furthermore, it yields more accurate quality estimates and facilitates diagnostic insights into individual rater performance, offering a robust foundation for informed decision-making in AI output assessment.