expert review

This study addresses the challenge of low-quality bug reports in crowdsourced testing, which impose substantial review burdens on developers and lack effective mechanisms to improve tester performance. The authors propose a large language model–based multi-agent evaluation framework that automatically assesses reports along three dimensions—textuality, sufficiency, and competitiveness—and integrates actionable feedback into human workflows. Through a four-phase controlled experiment combined with mixed-methods analysis, they provide the first empirical evidence that evaluative agents not only serve as post-hoc adjudicators but also function as in-process feedback sources, significantly enhancing the quality of report revisions, improving first-submission performance in subsequent tasks, and facilitating cross-application knowledge transfer. User studies further confirm the intelligibility and practical utility of the generated feedback.

This work addresses a critical limitation in existing cloud skill testing, which focuses solely on task success rates and fails to reveal uncovered behaviors, leaving test adequacy unquantifiable. To bridge this gap, the study introduces the first formal definition of test coverage units, coverage relationships, and a computational pipeline for cloud skills. It proposes a coverage evaluation method grounded in natural language skill packages: by parsing user prompts and initial resource states, the approach reconstructs operational obligations and models workflow context to establish an end-to-end measurement pipeline. Integrating model-assisted candidate generation with expert review, the framework produces auditable coverage reports and closes the loop by mapping coverage gaps to source-level test improvement recommendations. Empirical results demonstrate that this methodology enables quantitative assessment of test coverage for production-grade cloud skills, substantially enhancing their reliability and observability.

Fully automated peer review lacks accountability and is ill-suited for contexts requiring human oversight. This work proposes a “judgment-guided expansion” paradigm, wherein human reviewers provide evaluative claims that the system expands into complete review texts through a structured generate–verify–refine pipeline. We formalize this task for the first time, establishing a collaborative review framework that balances automation with accountability. To support robust evaluation, we introduce candidate-set calibration and a large-scale assessment methodology grounded in conformal prediction. Experimental results demonstrate that conformal prediction effectively balances candidate set size against target coverage, offering both methodological foundations and empirical validation for building trustworthy collaborative peer review systems.

This work proposes a localized multi-agent feedback system based on small open-source large language models (LLMs), addressing the limitations of conventional feedback systems that treat feedback as static and unidirectional, thereby failing to support personalized, interactive formative guidance. The system uniquely implements fully interactive feedback through a multi-agent architecture featuring pedagogically oriented feedback generation, an LLM-as-a-judge regeneration mechanism guided by human-aligned criteria, and a context-aware reflexive tool-calling agent enabling students to ask follow-up questions and engage in dynamic dialogue. Experimental results demonstrate that criterion-guided regeneration significantly enhances feedback quality, and the interactive agents achieve performance comparable to state-of-the-art closed-source models in both efficiency and quality. Deployment in real classroom settings further validates the system’s effectiveness in guiding student inquiry.

This work addresses the widespread absence of systematic quality assurance mechanisms in enterprise deployments of generative AI systems, which hinders the establishment of organizational trust. To bridge this gap, the paper proposes a novel four-stage quality assurance framework centered on domain experts, encompassing structured specification definition, system construction, expert-driven testing and validation, and continuous production monitoring. For the first time, this approach deeply integrates domain expertise throughout the entire generative AI engineering lifecycle. By doing so, it effectively reconciles the capabilities of generative AI with organizational trust requirements, ensuring authoritative expert oversight and high-quality outputs across diverse application scenarios, thereby significantly enhancing enterprises’ capacity for trustworthy deployment of generative AI systems.

This study addresses the limitations of traditional manual user profiling—high cost and poor scalability—and the unreliability and lack of iterative refinement in existing single-pass large language model (LLM)-based approaches. To overcome these challenges, the authors propose PerGent, a novel method that introduces, for the first time in an industrial setting, a multi-agent collaborative framework comprising three LLM-based agents: a generator, a critic, and a coordinator. By integrating structured and unstructured external data sources such as interviews and surveys, PerGent enables multiple rounds of critique-and-refinement iterations to progressively enhance profile quality. Evaluated in a real-world deployment at Kinaxis, the method achieved a 96.9% expert approval rate, significantly outperforming all baseline methods by not only accurately reproducing expert-derived content but also generating substantial high-value supplementary insights.

This study addresses the limitations of existing AI evaluation methods, which often fail to align with real-world user needs, contextual nuances, and local policies, while manual assessment remains difficult to scale. To bridge this gap, the authors propose an auditable and iterative, context-aware evaluation framework that integrates persona-driven test case generation, domain-specific scoring rubrics, and a hybrid adjudication mechanism combining human reviewers and LLM-based judges. Automated scoring is activated only when sufficient agreement between LLM judgments and human annotations is achieved. A three-week pilot across four organizations involving 108 annotated question-answer pairs demonstrates that the approach effectively balances policy alignment with scalable automation, enabling reliable end-to-end evaluation of AI systems.

This work addresses the resource-intensive bottleneck in ontology engineering caused by poor communication between domain experts and ontology engineers during competency question (CQ) elicitation. The authors propose a semi-automated workflow that integrates large language models (LLMs) into an expert-in-the-loop process: initial CQs are automatically generated from requirement documents, then iteratively refined by the LLM based on expert feedback provided through an interactive platform until consensus is reached. This approach introduces, for the first time, an iterative CQ generation mechanism combining LLM capabilities with expert collaboration, complemented by a provenance model that fully traces the CQ lifecycle to ensure transparency and reproducibility. Evaluations in real-world scientific data and cultural heritage scenarios demonstrate that the method significantly accelerates requirements engineering and enhances CQ acceptability, relevance, and usability for domain experts.

This study investigates how human-in-the-loop (HITL) feedback influences users’ perceptions of system accuracy and trust, highlighting the critical moderating role of task subjectivity. Through three controlled user experiments that systematically differentiate between objective and subjective task contexts, the research analyzes behavioral measures to assess the effects of feedback interaction. Findings reveal that in objective tasks, providing feedback significantly diminishes users’ trust in and perceived accuracy of the system, whereas this negative effect vanishes in subjective tasks. These results underscore task type as a pivotal factor shaping human–AI trust dynamics and offer important theoretical grounding and practical guidance for the design of HITL systems.

Existing skill generation methods suffer from insufficient domain alignment and an inability to automatically refine skills based on execution failures after deployment, leading to performance plateaus. This work proposes the first end-to-end closed-loop framework that enables skill self-evolution: it first synthesizes initial skills by integrating domain knowledge with historical service tickets, then employs a three-stage pipeline—comprising a failure analyzer, a skill diagnoser, and an optimizer—to automatically identify deficiencies and iteratively rewrite skills. Experiments on 1,883 tickets spanning 3,737 tasks demonstrate that the generated initial skills significantly outperform general-purpose baselines, and the self-evolution process consistently enhances skills regardless of their starting quality, ultimately surpassing even those manually crafted by human experts.