baseline evaluation

Current human baselines in large language model evaluation lack methodological rigor and transparency, undermining the validity of claims such as “superhuman performance.” Method: This paper pioneers the systematic integration of classical measurement theory into AI evaluation, establishing a comprehensive theoretical framework spanning human baseline design, execution, and reporting. It introduces an actionable quality assessment system and a standardized checklist, derived via meta-review–driven framework development, structured checklist design, and empirical systematic auditing. Contribution/Results: Applying this framework to diagnose 115 human baseline studies, we identify pervasive methodological flaws. The resulting open-source audit tool significantly enhances reproducibility, comparability, and accountability in AI evaluation. By grounding benchmarking practice in psychometric principles, our work provides a rigorous methodological foundation for scientifically credible model capability assessment.

This study identifies systematic inconsistencies in the implementation of machine learning evaluation metrics across mainstream programming languages—Python, R, and MATLAB—spanning ten task categories: classification, regression, clustering, statistical testing, image segmentation, and image-to-image translation. Through the first large-scale, cross-platform empirical analysis, we quantitatively assess consistency across 100+ metrics. Results reveal that 36 metrics—including Accuracy, AUC, and MAE—are robust across implementations, whereas critical metrics such as Precision, F1-score, IoU, and Within-Cluster Sum of Squares (WCSS) exhibit substantial discrepancies. To address this, we propose the first comprehensive, task-agnostic standardization roadmap for ML evaluation, accompanied by a curated recommendation list. This work provides both theoretical foundations and practical guidelines to enhance cross-platform reproducibility and result reliability in ML research and deployment.

Existing multilingual, multitask NLP evaluation benchmarks lack statistically reliable metrics due to unquantified uncertainty from both model stochasticity and data sampling variability. Method: We propose a resampling framework that jointly models dual sources of variation—model randomness (e.g., weight initialization, training dynamics) and data sampling variability—using bootstrap to construct empirical sampling distributions for standard metrics (e.g., accuracy, BLEU, F1). This enables principled confidence interval estimation for key statistics including means, medians, pairwise model differences, and rankings. Contribution/Results: Evaluated on multilingual question answering, machine translation, and named entity recognition, our approach significantly improves the precision of performance fluctuation characterization. It enhances comparability and reproducibility across models and languages, offering an interpretable, reproducible paradigm for uncertainty quantification in NLP benchmarking.

This study addresses a critical gap in current AI evaluation methodologies, which often overlook the impact of low-resource deployment conditions—such as noisy inputs, limited hardware capabilities, and unstable network connectivity—on system usability. The work proposes a novel evaluation framework that treats the deployed system as the unit of assessment, integrating task performance with real-world deployment contexts across multiple dimensions. Departing from conventional leaderboard-based approaches, the framework tailors evaluation criteria to specific application categories and introduces a standardized reporting system comprising benchmark cards, deployment profiles, and failure-handling mechanisms. By balancing comparability with contextual sensitivity, this approach provides policymakers and practitioners with clear, actionable insights for informed AI deployment decisions.

Current evaluation of long-form question answering systems predominantly relies on human pairwise preference judgments, which often fail to capture the nuanced, expert-level assessment of in-depth research report quality. This work systematically examines the applicability and limitations of such meta-evaluation approaches in scientific QA using the ScholarQA-CS2 benchmark. The study finds that pairwise preferences are suitable only for system-level comparisons, whereas metric-level evaluation requires explicit dimension-wise annotations combined with domain-expert review. It identifies subjectivity as a central challenge and proposes a set of meta-evaluation design guidelines aligned with expert expectations, offering practical recommendations for future evaluation frameworks, annotator expertise matching, and reporting practices in deep research-oriented QA systems.

Current language model benchmarks often suffer from coarse-grained metadata, making it difficult to accurately assess their coverage of capabilities that matter to users. To address this limitation, this work proposes a fine-grained retrieval system based on natural language queries that precisely identifies evaluation items relevant to real-world usage scenarios across 20 mainstream benchmarks. For the first time, the system leverages interpretable retrieval evidence to expose gaps between benchmark content and user intent. It further enables transparent validation of benchmark validity through human evaluation combined with analyses of content validity and construct validity. Human assessment confirms that the method achieves high retrieval precision and effectively uncovers issues such as insufficient capability coverage or unstable scoring.

Current benchmarks for medical large language models struggle to accurately predict real-world deployment performance, primarily because they fail to explicitly model the implicit assumptions underlying user–model interactions. This work proposes a conceptual distinction between “task assumptions,” which can be validated from dialogue data, and “outcome assumptions,” which require integration of outcome data and behavioral studies. To systematically evaluate how these assumptions contribute to the benchmark–deployment performance gap, the authors introduce BenchmarkCards—a novel framework coupled with a phased evaluation protocol. Retrospective clinical trial analyses and behavioral studies in case applications demonstrate that both types of assumptions contribute comparably to performance discrepancies, thereby validating the framework’s operational feasibility and practical utility, and significantly enhancing benchmark validity.

This work addresses the high cost and limited comparability of language model evaluations amid rapid iterations of models and benchmark datasets. To this end, the authors propose a dynamic, comparable evaluation framework grounded in multidimensional Item Response Theory (IRT). By fixing calibrated item parameters and incorporating a small set of anchor items—requiring only 100 questions per dataset—the framework enables efficient and comparable assessment of new models or datasets. Validated across more than 400 models, the approach predicts full-evaluation performance with an error margin of just 2–3 percentage points and achieves a Spearman rank correlation of at least 0.9, thereby substantially reducing evaluation overhead while preserving score comparability across time and tasks.

This study addresses a critical gap in existing tool-calling evaluation benchmarks: the lack of validation of the evaluators themselves, which risks conflating assessment artifacts with agents’ true capabilities. Through a systematic audit of four prominent benchmarks—BFCL v4, τ2-Bench, LiveMCPBench, and MCP-Atlas—the authors conduct expert review of 496 tasks, replicate experiments, and perform trajectory-level analysis, revealing an 18.5% disagreement rate between automated evaluators and human judgment. Notably, LiveMCPBench exhibits a score variance of up to 18.9 percentage points upon re-evaluation, sufficient to overturn leaderboard rankings. To address these issues, the work introduces the first unified taxonomy of tool-calling evaluation failures, advocates for distinct measurement of tool invocation, task completion, and result verification, and releases Tool-Veritas—a configurable benchmark—and Harness Lab, an open-source evaluation platform.