llm-as-judge evaluation

This work addresses the limitations of traditional static benchmarks—prone to saturation, contamination, and high updating costs—and the susceptibility of existing large language model (LLM) auto-scoring methods to prompt sensitivity and bias. It proposes the first three-stage framework that evaluates LLMs’ *benchmark design capability* rather than merely their question-answering performance. The approach leverages structured domain cards for extraction, quota-based multi-model collaborative item generation, and scoring via precise, numerical, and symbolic verifiers combined with psychometric analysis. From nine domains, it generates 16.7K items (retaining 15K core items) and constructs a designer–responder matrix with 152K scoring records. Empirical results reveal only a moderate correlation between design and answering abilities (Spearman ρ ≈ 0.37) and a strong negative association between invalid items and discrimination (r ≈ −0.62), demonstrating the framework’s effectiveness for scalable, cross-modal, and multilingual benchmark auditing.

This work addresses the limitations of existing automatic evaluation methods for large language model (LLM) outputs, which often rely on reference texts and exhibit limited generalizability, thereby struggling to accurately assess the quality and relevance of generated content. The authors propose a reference-free, domain-agnostic automated evaluation framework that leverages pairwise comparisons among multiple LLMs, integrated with an Elo rating system to produce stable and interpretable rankings. A tunable consistency threshold is introduced to balance evaluation confidence against coverage. Evaluated on scientific abstract quality assessment, the method yields rankings that align closely with expert judgments, significantly reducing the need for manual evaluation while demonstrating near-expert assessment capability.

This work investigates whether LLM-generated evaluation benchmarks satisfy three core requirements: phenomenon specificity, shortcut resistance, and high difficulty. The authors construct LLM-synthesized variants of CondaQA (for negation reasoning) and DROP (for quantitative reasoning) via prompt engineering, then rigorously evaluate their validity and difficulty through human annotation consistency checks and cross-model performance comparisons. Results reveal— for the first time—that although LLM-synthesized data adhere to annotation guidelines and cost only a fraction of crowdsourcing, they exhibit a 12.7% average reduction in challenge level for mainstream LLMs and are significantly more susceptible to superficial pattern exploitation. This study uncovers a systemic difficulty decay in current LLM-generated benchmarks, risking inflated estimates of model capability. It provides critical empirical evidence and methodological insights for developing trustworthy, rigorously validated evaluation benchmarks in language model assessment.

Existing language model evaluation benchmarks suffer from overly abstract criteria, coarse granularity, and coverage bias. To address these limitations, we propose BiGGen Bench—the first generative evaluation benchmark targeting nine fine-grained capabilities (e.g., reasoning consistency, factual controllability) across 77 diverse tasks. Our method introduces instance-level dynamic evaluation criteria and a language model self-assessment paradigm, enabling capability disentanglement and balanced assessment via collaborative scoring by multiple evaluator LMs, task-aware prompt engineering, and a structured evaluation protocol. We further develop an extensible, reproducible, and fully open-source automated evaluation framework. Comprehensive evaluation of 103 state-of-the-art models reveals critical capability bottlenecks across dimensions. All code, data, and results are publicly released.

This work addresses the proliferation of large language model (LLM) evaluation benchmarks, which has outpaced systematic assessment of their intrinsic quality. To this end, we propose Benchmark², a novel framework that establishes the first quantitative methodology for evaluating the reliability and validity of LLM benchmarks through three complementary metrics: cross-benchmark ranking consistency, discriminability score, and capability alignment bias. Empirical evaluation across 15 benchmarks and 11 LLMs demonstrates that Benchmark² not only reveals substantial quality disparities among existing benchmarks but also enables the construction of streamlined test sets that maintain high evaluative performance while significantly reducing assessment scale.

This work addresses the lack of reproducible and calibratable prompt engineering pipelines for evidence synthesis tasks in current large language models (LLMs). It proposes an innovative workflow that decouples scientific task specifications from prompting frameworks for the first time, leveraging annotated data and explicit metrics to drive prompt optimization. The approach operationalizes the entire pipeline into artifacts using the DSPy and GEPA toolchains, employing a small student model to execute tasks while a larger reflection model guides iterative refinement. Supporting structured task definitions, metric-driven search, and cross-framework portability, the method demonstrates strong compilability and artifact consistency in title and abstract screening tasks. Empirical validation further reveals the critical impact of optimization budgets on small-model performance, significantly enhancing the reliability and transparency of LLM-based applications.

This study addresses a critical limitation in current single-prompt accuracy evaluations of language models, which overlook key reliability issues such as miscalibration, parsing failures, and prompt sensitivity. The authors propose a systematic reliability auditing framework driven by multiple prompt variants to assess 15 open-source models across dimensions including accuracy, probability calibration, linguistic confidence, parse success rate, and robustness to prompt perturbations. By introducing techniques such as label-normalized expected calibration error (ECE), chain-of-thought reasoning combined with first-token evaluators, confidence parsing, and perturbation propagation analysis, the work demonstrates how evaluation design profoundly influences conclusions. Findings reveal that assessment methodology can inflate accuracy estimates by up to 88%, linguistic confidence is consistently overestimated, and model size shows no stable correlation with prompt robustness (correlation coefficients ranging from –0.244 to 0.474), underscoring the necessity of transparent reporting of evaluation protocols.

This work addresses the limitations of existing representation engineering approaches, which rely on synthetic data and suffer from irreproducible evaluations and susceptibility to superficial patterns. The authors construct the first large-scale, multi-source aligned capability representation framework grounded in real-world benchmarks, curated from over 10,000 academic papers and hundreds of public datasets, spanning 94 distinct capabilities. This framework enables cross-benchmark aggregation of capability vectors and transferable evaluation, effectively mitigating bias from any single data source. Experiments across 12 large language models reveal that benchmark-pooled capability vectors exhibit stable clustering structures; differential mean achieves the best performance in 10 models, while logistic regression outperforms others across the greatest number of capability–model combinations, underscoring the critical influence of both evaluation dimensions and readout methodologies.

This study addresses underexamined translation errors in current machine translation benchmarks that may compromise the reliability and comparability of multilingual large language model (LLM) evaluations. It presents the first systematic quantification of the isolated impact of target-side translation errors on multilingual LLM assessment outcomes. The approach leverages an LLM-based evaluator to generate MQM-style error annotations, integrates the xCOMET-XXL quality estimation model, and employs controlled variable analysis while holding the correctness of English source prompts constant. Findings indicate that although automatically generated error annotations exhibit discrepancies compared to human judgments, translation errors nonetheless induce a statistically significant drop in model accuracy. This result validates the efficacy of automated error localization methods when applied to real-world translation benchmarks.