rubric design

Behavioral profiling (BP) annotation is challenging to automate due to its multidimensional, multilingual nature, and conventional task-level evaluation obscures underlying skill heterogeneity. This work proposes a novel “skill feasibility” paradigm, decomposing BP annotation into 14 operationalizable annotation skills and implementing a schema-guided, skill-document-driven pipeline. Evaluation over a 300-instance validation set—through two rounds of testing involving human annotators and large language models (GPT-5.4 and three open-source models)—reveals a “shared categorization, independent execution” pattern: humans and GPT exhibit high agreement at the skill level but diverge in instance-level execution. The study identifies five directly feasible skills, four recoverable via relabeling, and five structurally undefined. GPT-5.4 demonstrates reliable performance on feasible skills (accuracy = 0.678, κ = 0.665, weighted F1 = 0.695), whereas open-source models primarily fail in translating schemas into executable skills.

Automated scoring systems in large-scale educational assessments suffer from a critical lack of transparency and interpretability. Method: This paper proposes FGTI—four principled interpretability criteria (Faithfulness, Groundedness, Traceability, Interchangeability)—to establish the first systematic theoretical framework for explainable automated scoring, and releases the benchmark model AnalyticScore. It leverages large language models to extract semantically explicit, human-identifiable features from student responses, constructs human-understandable feature vectors, and applies ordinal logistic regression for interpretable score prediction. Results: On the ASAP-SAS 10-task dataset, AnalyticScore achieves a mean quadratic weighted kappa (QWK) only 0.06 lower than the best black-box model, while its feature extraction exhibits strong agreement with human annotations (Spearman’s ρ > 0.92), significantly enhancing the trustworthiness and pedagogical validity of AI-based scoring.

This study addresses the limited reliability of large language models (LLMs) as evaluators in high-difficulty pairwise comparison tasks—particularly those involving factual consistency in long documents, mathematical reasoning, and code correctness. We propose an agent-based evaluation framework endowed with external verification capabilities: it dynamically retrieves real-time, verifiable evidence via explicit web search and code execution tools, and integrates structured prompt engineering to mitigate LLMs’ inherent knowledge biases and hallucinations. Experiments across multiple challenging benchmarks demonstrate substantial improvements in evaluator consistency and accuracy. Moreover, we identify strong sensitivity of evaluation performance to tool invocation strategies and prompt design. Our work advances non-saturating, empirically verifiable LLM evaluation paradigms and offers a novel pathway toward building trustworthy AI feedback systems.

This study addresses the limitations of existing AI evaluation methods, which often fail to align with real-world user needs, contextual nuances, and local policies, while manual assessment remains difficult to scale. To bridge this gap, the authors propose an auditable and iterative, context-aware evaluation framework that integrates persona-driven test case generation, domain-specific scoring rubrics, and a hybrid adjudication mechanism combining human reviewers and LLM-based judges. Automated scoring is activated only when sufficient agreement between LLM judgments and human annotations is achieved. A three-week pilot across four organizations involving 108 annotated question-answer pairs demonstrates that the approach effectively balances policy alignment with scalable automation, enabling reliable end-to-end evaluation of AI systems.

Large language models (LLMs) exhibit poor reproducibility in text annotation tasks due to sensitivity to minor prompt perturbations, yet no standardized metric exists for quantifying prompt stability. To address this, we systematically adapt inter-annotator agreement principles from coding reliability research to prompt engineering, introducing the Prompt Stability Score (PSS)—a unified, computationally tractable metric for stability assessment. Our method integrates multi-prompt sampling, batched LLM inference, consistency analysis via Cohen’s and Fleiss’ Kappa, and an automated Python evaluation framework (open-sourced as PromptStability). Empirical validation across six benchmark datasets and twelve annotation task types—encompassing over 150,000 samples—demonstrates PSS’s effectiveness in precisely identifying low-stability prompting configurations. This work establishes the first standardized diagnostic paradigm for evaluating prompt robustness, thereby enabling reproducible, interpretable, and empirically grounded prompt engineering practices.

This study addresses the lack of empirical validation for existing retrieval-augmented generation (RAG) evaluation metrics in real-world scenarios. Leveraging a human-annotated commercial question-answering dataset, it presents the first systematic comparison of prominent metrics from four major evaluation frameworks—Ragas, DeepEval, RAGChecker, and Opik. Through correlation analyses against human judgments and traditional metrics such as recall, the work reveals that most automatic metrics exhibit weak alignment with human assessments. These findings not only highlight significant limitations in current RAG evaluation methodologies when applied to practical settings but also provide empirical grounding and actionable directions for developing more reliable, real-world-oriented evaluation approaches.

This work addresses the limited reliability of structured security artifacts—such as KQL queries and MITRE ATT&CK mappings—generated by large language models, which often fall short of production-grade requirements. To bridge this gap, the authors propose a lightweight verification framework that shifts the focus of quality assurance from generation to validation. The core innovations include a hybrid testing strategy integrating test-driven generation, deterministic program verification, and semantic evaluation by large language models, alongside an interpretable judging mechanism distilled from expert decision distributions. Deployed in Microsoft Sentinel’s production environment, the framework significantly enhances the reliability of three critical types of security artifacts, establishing professional-grade and scalable validation standards.

This study addresses the reproducibility crisis in AI evaluation, which stems from subjective biases in human annotation and limited data leading to unstable results. The authors propose a multilevel bootstrapping framework that, for the first time, leverages large-scale human rating data—complete with persistent annotator identifiers—to model annotator behavior and systematically quantify the joint impact of the number of items (N) and the number of ratings per item (K) on evaluation reproducibility. Through multilevel bootstrap resampling, variance modeling, and significance analysis, the work uncovers the trade-offs inherent in N–K configurations and derives optimal strategies for reliable assessment design. This approach provides a methodological foundation for establishing robust, data-driven paradigms in AI evaluation.

Current evaluation practices for text-to-image generation models predominantly rely on uniform annotation protocols that overlook the intrinsic differences among distinct assessment skills, resulting in unreliable evaluation signals. This work proposes a “skill-aligned annotation” strategy that systematically tailors annotation methodologies to the specific characteristics of each evaluation skill, thereby establishing a scalable, fine-grained, and spatially grounded automatic evaluation pipeline. Through comparative experiments, inter-annotator consistency analysis, and spatial grounding feedback, the proposed approach significantly enhances both annotator agreement and cross-model evaluation stability. Crucially, it achieves more reliable and efficient assessment of image generation quality without increasing the burden of manual annotation.

This study addresses whether the outputs of small language models (SLMs) in psychometric tasks stem from genuine semantic reasoning or are primarily driven by artifacts of prompt formulation. The authors propose the first diagnostic framework capable of disentangling the influence of such prompt artifacts, systematically manipulating role framing, instructions, item content, and option labels while employing controlled experiments and variance decomposition techniques to quantify the relative contributions of semantic signals versus prompt-induced artifacts. Findings reveal that prompt artifacts frequently dominate model responses, substantially undermining their psychometric validity. The proposed framework not only effectively identifies these confounding influences but also offers a novel pathway for evaluating and enhancing the semantic comprehension capabilities of large language models.