cross-dataset evaluation

This work addresses the fundamental lack of fairness and robustness in evaluating model transferability across domains. We propose the first systematic, standardized benchmarking framework for assessing cross-domain transfer capability. Our method introduces a unified multi-source-domain–target-domain evaluation protocol, encompassing diverse transfer tasks and perturbation-robustness analysis, and adopts head-training (i.e., linear-probe fine-tuning) as the consistent evaluation paradigm. Empirical analysis reveals significant performance discrepancies among existing transferability metrics under varying experimental settings, undermining their reliability. Our framework substantially improves assessment fidelity, yielding an average 3.5% gain in transfer performance under standard head-training configurations. To foster reproducibility and rigorous comparison, we fully open-source all code, datasets, and evaluation pipelines—establishing a new, standardized paradigm for transferability measurement.

Existing synthetic data evaluation lacks unified, transferable quantitative metrics. This paper proposes a novel evaluation framework grounded in generalized cross-validation (GCV) and domain transfer learning. It constructs a cross-dataset performance matrix and defines two core metrics: *fidelity*, quantifying distributional similarity between synthetic and real data; and *generalization coverage*, measuring the task-transfer capability of synthetic data across diverse real-world source domains. The framework is model-agnostic and enables normalized, comparative evaluation of detectors such as YOLOv5s across heterogeneous datasets—including Virtual KITTI, KITTI, and BDD100K. Experiments demonstrate that the method effectively quantifies synthetic data quality, significantly enhancing evaluation generality, comparability, and utility for model optimization. It establishes a scalable, reproducible, and standardized evaluation paradigm for synthetic data development.

It remains unclear whether a large language model’s (LLM) out-of-distribution (OOD) generalization performance on a single OOD dataset reliably reflects its true robustness under diverse distribution shifts. Method: We introduce the first systematic multi-OOD evaluation framework, leveraging partial correlation analysis to quantify inter-test-set generalization correlations while controlling for in-domain performance confounds. Results: Experiments across OLMo2 and OPT models reveal that OOD generalization exhibits no consistent positive or negative cross-test-set correlations; instead, correlation patterns are highly dependent on specific model–test-set pairings. This demonstrates the inadequacy of single-OOD benchmarks for assessing robust generalization. Our findings provide theoretical grounding and methodological support for developing more reliable, reproducible, and comprehensive generalization evaluation protocols—highlighting the necessity of multi-OOD assessment to uncover latent failure modes and ensure meaningful robustness estimation.

How can NLP models’ cross-domain performance be reliably evaluated and ranked without labeled data? This paper proposes a two-stage unsupervised evaluation framework: first, jointly modeling prediction errors using four base classifiers and multiple large language models (LLMs); second, assessing ranking reliability by measuring alignment between inferred error patterns and domain discrepancies. Experiments on multi-domain benchmarks—GeoOLID and Amazon Reviews—demonstrate that ranking stability and correlation with true accuracy significantly improve when predicted error distributions closely match actual failure modes. Compared to distribution-shift–based or zero-shot alternatives, our approach substantially enhances robustness and interpretability of cross-domain performance ranking. Crucially, it is the first work to systematically characterize the applicability boundaries and key determinants—namely, error-pattern fidelity and domain-difference alignment—of label-free performance estimation.

Existing multilingual, multitask NLP evaluation benchmarks lack statistically reliable metrics due to unquantified uncertainty from both model stochasticity and data sampling variability. Method: We propose a resampling framework that jointly models dual sources of variation—model randomness (e.g., weight initialization, training dynamics) and data sampling variability—using bootstrap to construct empirical sampling distributions for standard metrics (e.g., accuracy, BLEU, F1). This enables principled confidence interval estimation for key statistics including means, medians, pairwise model differences, and rankings. Contribution/Results: Evaluated on multilingual question answering, machine translation, and named entity recognition, our approach significantly improves the precision of performance fluctuation characterization. It enhances comparability and reproducibility across models and languages, offering an interpretable, reproducible paradigm for uncertainty quantification in NLP benchmarking.

This work addresses the critical challenge of evaluating model generalization in high-stakes scenarios with scarce labels, where existing methods lack reliable, label-free metrics for pre-deployment model selection and post-deployment performance monitoring. To bridge this gap, the study introduces, for the first time, the internal causal circuit mechanisms of Vision Transformers into generalization assessment, proposing two novel unsupervised metrics: Dependency Depth Bias and Circuit Shift Score. The former quantifies depth-wise biases in representational dependency structures, while the latter measures changes in circuit stability under distribution shifts. Extensive experiments across diverse tasks demonstrate that these metrics achieve substantially higher correlations with true generalization performance—improving by 13.4% and 34.1% on average over current approaches—thereby significantly enhancing the reliability of generalization prediction without requiring ground-truth labels.

Existing evaluation methods struggle to disentangle overall performance gains in source languages from genuine cross-lingual transfer capabilities in multilingual models. To address this limitation, this work proposes the Hardness-Adjusted Transfer (HAT) score, which isolates source-language performance to more accurately quantify transfer effectiveness from high-resource to low-resource languages. Leveraging HAT, we conduct a large-scale empirical analysis across 20 language models and three major multilingual benchmarks, revealing—for the first time—that small models retain meaningful transfer capacity, that scaling model size yields diminishing returns in transfer gains, and that overall cross-lingual transfer capability has steadily improved over time.

Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under fixed inputs, such as truthfulness, instruction following, or pairwise preference, and these scores are often used to support claims about deployed alignment. This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone. Alignment claims should instead be indexed to the level at which evidence is collected: model-level, response-level, interaction-level, or deployment-level. Two studies support this position. First, a structured audit of eleven alignment benchmarks, extended to a sixteen-benchmark corpus, dual-coded against an eight-dimension rubric with Cohen's kappa = 0.87, finds that user-facing verification support is absent across every benchmark examined, while process steerability is nearly absent. The few interactional benchmarks identified, including tau-bench, CURATe, Rifts, and Common Ground, remain fragmented in coverage, and benchmark construction rather than data source determines what is measured. Second, a blinded cross-model stress test using 180 transcripts across three frontier models and four scaffolds finds that the same verification scaffold raises one model's verification support to ceiling while leaving another categorically unchanged. This shows that scaffold efficacy is model-dependent and that the gap identified by the audit cannot be closed at the model level alone. We propose a system-level evaluation agenda: alignment profiles instead of single scores, fixed-scaffolding protocols for comparable interactional evaluation, and reporting templates that make the inferential distance between evaluation evidence and deployment claims explicit.

This study addresses the underexplored engineering challenges in existing machine learning evaluation frameworks, where operational issues and their root causes have lacked systematic investigation. To bridge this gap, the work formally establishes evaluation engineering as a distinct research direction within software engineering. Through an empirical analysis of 57 frameworks and a comprehensive categorization of 16,560 reported issues across a newly proposed five-stage workflow model, the study reveals that 41.4% of problems originate in the specification phase, while 61.7% of classified issues stem from missing functionality, inadequate documentation, and insufficient input validation. The findings yield a structured taxonomy of evaluation-related problems and provide empirical evidence to inform the design and improvement of robust evaluation systems.