llm evidence scoring

Accurately extracting UK Research Excellence Framework (REF) ratings (1*–4*) from noisy, unstructured text containing missing or invalid values presents a significant challenge, requiring large language models (LLMs) to output only normalized integers (1–4) or a designated missing-value indicator (−1). To address this, this work introduces the first standardized prompt engineering benchmark for complex numerical extraction tasks, accompanied by a publicly available dataset of 1,446 short texts with gold-standard annotations. By integrating semantic understanding with explicit rule-based constraints, an initial prompting strategy achieves 72.6% accuracy. The study clarifies the definition of valid ratings and formalizes a mechanism for handling missing data, thereby advancing research into LLMs’ numerical reasoning and instruction-following capabilities, with the aim of fostering community-driven improvements in structured information extraction from noisy textual sources.

This study addresses the challenge of error-prone manual verification of tables, figures, and listings (TFLs) in clinical trial reports, which often fails to detect structural or logical inconsistencies. The authors propose PROVE, a novel framework that leverages large language models (LLMs) for semantic parsing and evidence tracing of TFL content, integrated with a programmable rule engine to perform deterministic numerical and logical validation against SDTM/ADaM standards. Designed as a multi-agent architecture, PROVE combines LLM-driven semantic understanding with rule-based checks to enable auditable, configurable automated cross-verification. The approach achieves 100% accuracy under exact label matching; when confronted with linguistic variations, LLM assistance boosts recall from 0.588 to 0.993 and F1 score from 0.735 to 0.996.

Current LLM misuse detection tools in online learning suffer from low reliability, lack of standardized evaluation criteria, and insufficient understanding of educational implications. Method: This study proposes actionable, interpretable criteria for identifying LLM-generated text and introduces a novel multidimensional detection paradigm based on fine-tuned GPT-4o, integrating statistical indicators—including anomalously high scores, readability metrics, and response latency. Results: The method achieves 80% accuracy (F1 = 0.78) on short-answer detection, substantially outperforming GPTZero (70%, F1 = 0.50); robustness is confirmed via human coding and comparative evaluation. Empirical analysis further reveals that students misusing LLMs exhibit abnormally elevated post-test accuracy—indicating superficial engagement and bypassing of deep learning processes. This work constitutes the first systematic integration of detection methodology, interpretable decision criteria, and pedagogical impact analysis, providing both a methodological framework and empirical foundation for AI governance in education.

Large language models (LLMs) used as evaluators suffer from estimation bias and noise due to insufficient specificity and sensitivity, undermining the statistical reliability of automated evaluation. Method: We propose the first practical framework that corrects such bias and constructs statistically rigorous confidence intervals. It introduces a plug-in bias-correction mechanism that jointly models uncertainty over both test and calibration sets, coupled with an adaptive sampling algorithm to optimize calibration sample allocation. Leveraging estimated specificity and sensitivity, the framework employs statistical inference to derive bias-corrected confidence intervals. Contribution/Results: Our approach significantly reduces both bias and variance in accuracy estimation. Extensive evaluation across multiple benchmarks demonstrates its robustness, reliability, and generalizability. The framework establishes a new paradigm for LLM-based automated evaluation—reproducible, interpretable, and statistically trustworthy.

Conventional LLM evaluation metrics exhibit saturation effects, failing to discern subtle differences—particularly in reasoning capabilities—among state-of-the-art models. Method: We propose the “Gibberish Technique”, a scalable evaluation augmentation paradigm that transforms original tasks into a family of progressively challenging, reasoning-oriented multiple-choice questions. This is achieved through semantics-preserving input perturbations and multi-level difficulty construction. Contribution/Results: The technique uncovers previously masked capability gradients between base models and specialized reasoning models—revealing distinctions invisible under standard benchmarks. Experiments across multiple mainstream LLMs demonstrate that our augmented evaluation significantly improves discriminative power, accurately characterizing hierarchical reasoning competencies. It establishes a more sensitive and diagnostically informative benchmark for LLM assessment, enabling fine-grained differentiation where traditional metrics fall short.

This study addresses the threat posed by the stochasticity of large language model (LLM) outputs to research reproducibility, demonstrating that randomness arises not only from sampling strategies but also from non-sampling factors such as silent model updates, numerical rounding, and expert routing in mixture-of-experts architectures. The work is the first to explicitly model LLM outputs as draws from a probability distribution and systematically evaluates the impact of various sources of randomness on downstream outcomes through regression analysis in a sentiment classification task. Empirical comparisons across multiple software and hardware environments—using both API-accessible and locally deployed open-source models—reveal substantial output variability even at zero temperature. Building on these findings, the paper proposes standardized reporting guidelines for research papers and reproduction packages to encourage the adoption of stricter LLM usage protocols within the community.

This work addresses the challenges of behavioral drift, miscalibrated uncertainty, and declining trustworthiness in large language models (LLMs) under continuous deployment, primarily caused by inadequate modeling and monitoring of temporal interactions and dynamic feedback. To tackle this, the paper introduces, for the first time, a sequential statistical inference framework tailored for trustworthy LLM deployment. Built upon dependent stochastic processes, the proposed framework integrates sequential hypothesis testing, change-point detection, calibration assessment, and fairness monitoring. This paradigm provides rigorous uncertainty guarantees in settings involving dependent interactions, repeated usage, and adaptive behavior, while enabling real-time monitoring and early warning for critical attributes such as hallucination rates, refusal patterns, and fairness violations. The approach significantly enhances the stability and reliability of LLMs operating in dynamic environments.

This study addresses the questionable diagnostic validity of automated scoring by large language models (LLMs), which often appears superficially comprehensive. Leveraging the JorGPT dataset, this work pioneers a diagnostic-validity perspective to deconstruct LLMs’ structured outputs. By integrating statistical analyses, including variance inflation factor (VIF) assessment, with natural language processing techniques, it systematically compares human and machine scoring across multidimensional score redundancy, misconception detection rates, and tone bias. The findings reveal systematic deficiencies in LLM scoring, notably high sub-dimension collinearity, low misconception detection rates, and an absence of severity moderation. Furthermore, the analysis demonstrates that LLMs remain reliable primarily within procedural knowledge domains. Ultimately, this research delineates critical boundaries requiring human oversight for human–AI collaborative assessment in higher education.

论文针对语言模型校准问题，提出在NLP研究中将校准作为评价模型的重要标准，通过已有数据和方法提升模型可信度。

This study addresses the instability of large language model (LLM) annotations under varying task designs, revealing substantial “instrument uncertainty” that confidence scores fail to mitigate. By evaluating seven LLMs across twelve task designs on a sample of 3,000 tweets, this work systematically quantifies the impact of prompt engineering on annotation outcomes using Fleiss’ and Cohen’s Kappa coefficients. The findings demonstrate that design variations inflate the variance of prevalence estimates by 76- to 110-fold, far exceeding typical inter-annotator disagreement among humans. Furthermore, this research establishes that comparing multiple task designs is the only effective approach for measuring such uncertainty, exposing critical limitations in existing evaluation methodologies and highlighting systematic biases introduced by both model selection and prompt design.