quantitative response evaluation

Existing evaluation methodologies for large language models (LLMs) suffer from insufficient generalization assessment, as static benchmarks fail to capture the continuously expanding capability boundaries of evolving LLMs. Method: We formally define “evaluation generalizability” and propose a four-dimensional analytical framework encompassing evaluation methodologies, datasets, evaluators, and metrics. Our approach innovatively integrates LLM-as-a-judge, dynamically updated datasets, capability-decoupled benchmark design, and a multidimensional meta-evaluation framework. Contribution: We establish a novel, capability-oriented, automated, and sustainably evolvable evaluation paradigm covering critical dimensions—including knowledge, reasoning, instruction following, multimodal understanding, and safety. Concurrently, we release an open-source, extensible GitHub “living review” repository—a community-maintained, versioned resource—to advance evaluation practice from static benchmarking toward dynamic, collaborative co-evolution.

This work addresses the proliferation of large language model (LLM) evaluation benchmarks, which has outpaced systematic assessment of their intrinsic quality. To this end, we propose Benchmark², a novel framework that establishes the first quantitative methodology for evaluating the reliability and validity of LLM benchmarks through three complementary metrics: cross-benchmark ranking consistency, discriminability score, and capability alignment bias. Empirical evaluation across 15 benchmarks and 11 LLMs demonstrates that Benchmark² not only reveals substantial quality disparities among existing benchmarks but also enables the construction of streamlined test sets that maintain high evaluative performance while significantly reducing assessment scale.

Current language model benchmarks often suffer from coarse-grained metadata, making it difficult to accurately assess their coverage of capabilities that matter to users. To address this limitation, this work proposes a fine-grained retrieval system based on natural language queries that precisely identifies evaluation items relevant to real-world usage scenarios across 20 mainstream benchmarks. For the first time, the system leverages interpretable retrieval evidence to expose gaps between benchmark content and user intent. It further enables transparent validation of benchmark validity through human evaluation combined with analyses of content validity and construct validity. Human assessment confirms that the method achieves high retrieval precision and effectively uncovers issues such as insufficient capability coverage or unstable scoring.

This work addresses the limitations of traditional static benchmarks—prone to saturation, contamination, and high updating costs—and the susceptibility of existing large language model (LLM) auto-scoring methods to prompt sensitivity and bias. It proposes the first three-stage framework that evaluates LLMs’ *benchmark design capability* rather than merely their question-answering performance. The approach leverages structured domain cards for extraction, quota-based multi-model collaborative item generation, and scoring via precise, numerical, and symbolic verifiers combined with psychometric analysis. From nine domains, it generates 16.7K items (retaining 15K core items) and constructs a designer–responder matrix with 152K scoring records. Empirical results reveal only a moderate correlation between design and answering abilities (Spearman ρ ≈ 0.37) and a strong negative association between invalid items and discrimination (r ≈ −0.62), demonstrating the framework’s effectiveness for scalable, cross-modal, and multilingual benchmark auditing.

This work addresses the limitations of existing representation engineering approaches, which rely on synthetic data and suffer from irreproducible evaluations and susceptibility to superficial patterns. The authors construct the first large-scale, multi-source aligned capability representation framework grounded in real-world benchmarks, curated from over 10,000 academic papers and hundreds of public datasets, spanning 94 distinct capabilities. This framework enables cross-benchmark aggregation of capability vectors and transferable evaluation, effectively mitigating bias from any single data source. Experiments across 12 large language models reveal that benchmark-pooled capability vectors exhibit stable clustering structures; differential mean achieves the best performance in 10 models, while logistic regression outperforms others across the greatest number of capability–model combinations, underscoring the critical influence of both evaluation dimensions and readout methodologies.

Traditional language model evaluation often conflates the ability to produce assessable responses with the correctness of those responses, thereby masking execution-level failure modes under aggregate accuracy metrics. This work proposes a two-tiered evaluation framework that disentangles scorer-agnostic execution states—such as termination, answer exposure, parseability, and output length—from scorer-dependent correctness judgments. By enforcing a fixed output budget, tracking multidimensional execution trajectories, formulate a verification mechanism driven by coverage auditing, the study systematically uncovers divergent execution behaviors across models on MATH and ARC-Challenge benchmarks. The analysis reveals that extended output lengths can mitigate certain failure modes and demonstrates that verification strategies substantially influence comparative accuracy outcomes.

Current evaluations of large language models (LLMs) on ill-defined tasks—such as complex instruction following and natural language-to-Mermaid sequence diagram generation—suffer from insufficient coverage, sensitivity to phrasing, incomparable metrics, and instability in LLM-based judging, thereby failing to yield reliable or diagnostic assessment signals. This work presents the first systematic analysis of confounding failure modes in such tasks, integrating case studies, failure mode categorization, and a multidimensional evaluation framework to demonstrate how existing benchmarks often conflate distinct error types, leading to distorted scores. Moving beyond monolithic aggregate metrics, the proposed approach delivers actionable, fine-grained insights that lay both theoretical and practical foundations for building more robust and interpretable evaluation systems.

This study addresses the lack of robustness in enterprise-grade large language models (LLMs) against minor input perturbations—a critical issue often overlooked by existing evaluations confined to academic settings. The authors present the first multidimensional perturbation benchmark tailored for enterprise applications, encompassing 11 perturbation types including textual edits, format variations (e.g., JSON/YAML), multilingual inputs, and instruction reordering. They systematically evaluate 11 prominent LLMs ranging from 4B to over 120B parameters. Results reveal that minor perturbations can degrade performance by up to 40 percentage points. Notably, model scale exhibits a nonlinear relationship with robustness: Mistral 3 8B consistently outperforms larger models, while Llama 3.1 8B performs worst, underscoring the pivotal roles of architecture and training data—and challenging the prevailing assumption that larger models are inherently more robust.

Current LLM evaluation benchmarks suffer from insufficient construct validity—particularly for abstract constructs such as “safety” and “robustness”—due to widespread construct–task misalignment across task design, phenomenon definition, and scoring metrics. Method: We systematically reviewed 445 benchmarks from top-tier conferences (ACL, EMNLP, NeurIPS), identifying eight recurrent validity threat patterns through expert-guided, systematic literature review. Contribution/Results: We propose, for the first time from a construct validity perspective, eight actionable benchmark design principles and an accompanying validation guideline. These provide a theoretical framework and empirical foundation for enhancing the scientific rigor and result reliability of LLM evaluation. Our work fills a critical methodological gap in LLM assessment by establishing a systematic validity verification paradigm, thereby shifting benchmark development from empirically driven practice toward validity-driven science.

Current evaluations of large language models for low- and medium-resource languages such as Icelandic heavily rely on unverified synthetic or machine-translated data, leading to unreliable assessment outcomes. This work presents the first systematic quantitative error analysis of data quality in such evaluation benchmarks, comparing human-authored or human-translated data against automatically generated counterparts across dimensions including linguistic accuracy and task fidelity. The study reveals that unverified data significantly deviates from human-curated references, substantially distorting model performance estimates. These findings underscore the critical importance of human validation in constructing trustworthy evaluation benchmarks and offer methodological guidance for robust assessment practices in low-resource language settings.