alignment evaluation

Conventional dimensionality reduction (DR) evaluation suffers from systematic bias due to the frequent adoption of highly correlated metrics, leading to overemphasis on specific structural properties. Method: We propose an empirically grounded metric redundancy reduction framework: first computing Pearson correlation matrices across diverse datasets and DR algorithms; then applying clustering to identify functionally redundant metric groups; and finally retaining only the most representative metric per group—replacing subjective, intent-driven metric selection with objective, behavior-based clustering. Contribution/Results: Our approach significantly improves cross-dataset and cross-algorithm stability of DR evaluations, effectively mitigating structural biases inherent in traditional assessment protocols. Experimental validation demonstrates enhanced reproducibility and generalizability, establishing a principled, data-driven framework for fair and robust comparative evaluation of DR methods.

Existing interpretability methods lack standardized data benchmarks and struggle to assess whether model explanations genuinely reflect the intrinsic structure of training data. Method: We propose a model-agnostic framework for evaluating model-data consistency, grounded in Rubin’s potential outcomes framework to construct a model-free, data-driven baseline. This baseline quantifies the true separative effect of each feature on binary classification tasks. Model explanations are then diagnosed by comparing feature importance rankings against this causal, data-derived baseline. Contribution/Results: Our approach efficiently detects when models deviate from fundamental data-generating mechanisms. It offers strong interpretability, low computational overhead, and cross-model applicability. To our knowledge, it is the first causally grounded, feature-effect-based tool for validating model-data consistency—providing a foundational method for trustworthy AI evaluation.

Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under fixed inputs, such as truthfulness, instruction following, or pairwise preference, and these scores are often used to support claims about deployed alignment. This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone. Alignment claims should instead be indexed to the level at which evidence is collected: model-level, response-level, interaction-level, or deployment-level. Two studies support this position. First, a structured audit of eleven alignment benchmarks, extended to a sixteen-benchmark corpus, dual-coded against an eight-dimension rubric with Cohen's kappa = 0.87, finds that user-facing verification support is absent across every benchmark examined, while process steerability is nearly absent. The few interactional benchmarks identified, including tau-bench, CURATe, Rifts, and Common Ground, remain fragmented in coverage, and benchmark construction rather than data source determines what is measured. Second, a blinded cross-model stress test using 180 transcripts across three frontier models and four scaffolds finds that the same verification scaffold raises one model's verification support to ceiling while leaving another categorically unchanged. This shows that scaffold efficacy is model-dependent and that the gap identified by the audit cannot be closed at the model level alone. We propose a system-level evaluation agenda: alignment profiles instead of single scores, fixed-scaffolding protocols for comparable interactional evaluation, and reporting templates that make the inferential distance between evaluation evidence and deployment claims explicit.

This work addresses the critical challenge of effectively comparing structural differences between two entity resolution (ER) clustering results in the absence of ground-truth labels. It proposes the Case Count Metric System (CCMS), which introduces and operationalizes, for the first time, a quantitative framework for four types of cluster transformations—preservation, merging, splitting, and overlapping—without requiring labeled data. By leveraging a cluster-set transformation analysis algorithm, CCMS enables fine-grained, unsupervised comparison of ER outcomes. Integrated with interactive analysis and visualization capabilities, the system has been successfully deployed in both academic and industrial settings, significantly enhancing the interpretability and efficiency of ER method evaluation and tuning.

In biomedical image segmentation validation, metrics such as the Hausdorff distance suffer from implementation inconsistencies across open-source toolkits, compromising benchmark reliability, introducing biomarker bias, and posing clinical deployment risks. To address this, we systematically evaluate 11 widely used toolkits and introduce, for the first time, a reference implementation based on high-fidelity 3D surface meshes. Our framework integrates real-world clinical data and a cross-platform consistency analysis. Statistical analysis reveals significant inter-tool variation in Hausdorff distance computations (p < 0.001), with interpolation strategy, boundary handling, and sampling density identified as primary sources of discrepancy. Based on these findings, we propose a reproducible and verifiable paradigm for distance-based evaluation, accompanied by standardized computational guidelines. This work substantially enhances the reliability, comparability, and clinical translatability of segmentation assessment.

Existing text evaluation metrics, despite exhibiting high correlation with human judgments, are vulnerable to strategic manipulation and lack robustness against irrelevant perturbations. This work introduces dual criteria—statistical alignment and strategic alignment—and formally defines strategic alignment for the first time, establishing a principled framework that encompasses human correlation, degradation sensitivity, and robustness to manipulation. Grounded in mutual information theory, the authors propose a unified metric design framework composed of four components: information measures, estimation methods, text representations, and prediction mechanisms. Empirical results demonstrate that strong correlation with human scores does not imply strategic robustness; the proposed metrics significantly enhance resistance to manipulation across peer review, summarization, and question-answering tasks while maintaining high agreement with human evaluations.

This study addresses the limitations of conventional AI alignment approaches, which often reduce organizational decision-making to a single objective while overlooking pluralistic values and procedural heterogeneity. The authors propose the concept of “process alignment,” which evaluates whether large language models weight information in ways consistent with an organization’s decision-making strategies, rather than focusing solely on output accuracy. Empirical analyses in two domains—European Court of Human Rights (ECHR) rulings and German credit decisions—reveal that process alignment strongly correlates with output accuracy in the ECHR context (r = 0.85) but not in credit decisions (r = 0.15), where external interventions also yield inconsistent effects, exposing underlying historical biases. The findings underscore the necessity of incorporating multi-perspective, process-level considerations into alignment evaluation and caution that high process alignment is both difficult to achieve and not universally desirable.

Existing segmentation evaluation metrics often lack transparency and modularity, making them ill-suited for diverse tasks such as transparent object, specular surface, or lesion segmentation. This work proposes a unified evaluation framework that decomposes metrics into five modular components: prediction representation, target extraction, target matching, score computation, and metric reporting. For the first time, it systematically analyzes the implicit assumptions and design limitations of mainstream binary segmentation metrics through this modular lens. The framework enables task-aware customization of evaluation protocols, reveals evolutionary trajectories among existing metrics, and is accompanied by an open-source toolkit. By offering a principled and interpretable foundation, this approach paves the way for developing more rational and adaptable segmentation evaluation methodologies.