conduct expert validation

This work addresses the challenge of accurately matching papers to reviewers in large-scale academic peer review, where existing approaches are limited by coarse similarity metrics or non-scalable manual annotations. The authors propose MERIT, a two-stage framework that first leverages large language models (LLMs) to generate reward signals based on fine-grained rubric-guided expertise criteria, then trains a 4B-parameter reviewer evaluator via reinforcement learning. In the second stage, the knowledge of this evaluator is distilled into an efficient embedding-based retriever to enable scalable reviewer assignment. This study is the first to formulate granular expertise matching as a supervised signal, achieving state-of-the-art performance on the LR-Bench and CMU Gold datasets. Notably, the specialized evaluator outperforms larger general-purpose LLMs on reviewer-paper fit classification tasks.

Novelty assessment in academic peer review remains a critical yet underexplored challenge. Method: This paper introduces the first large language model–based, structured novelty assessment framework. It emulates expert reviewer behavior via a three-stage automated pipeline: (1) structured extraction of submission content, (2) literature-aware retrieval and synthesis of related work, and (3) claim-level comparative reasoning—explicitly modeling independent claim verification and contextual inference. The method integrates analysis of large-scale human review corpora, literature-aware information extraction, and evidence-driven judgment techniques. Contribution/Results: Evaluated on 182 submissions to ICLR 2025, the framework achieves 86.5% alignment with human reviewers’ reasoning processes and 75.3% agreement on final novelty judgments—substantially outperforming existing baselines—while markedly improving assessment transparency and consistency.

Communication barriers between data scientists and domain experts arise from oversimplified, accuracy-centric model performance reporting, hindering shared understanding of model limitations and contextual applicability. Method: We propose a visualization-mediated model explanation framework grounded in human-computer interaction principles, participatory design, and visual narrative techniques. This yields the first domain-expert-oriented model communication guideline—emphasizing risk, trade-offs, and situational appropriateness rather than isolated metrics like accuracy. An iterative empirical study was conducted using regression models, incorporating structured expert feedback for evaluation. Contribution/Results: The framework significantly improves domain experts’ ability to identify model limitations, recognize inherent trade-offs, and proactively make context-driven adoption decisions. Its core innovation lies in repositioning visualization as an interdisciplinary consensus-building medium—shifting the paradigm from “metric reporting” to “collaborative understanding.”

This work addresses systematic limitations in existing creative quality alignment (CQA) datasets, particularly their inadequate modeling of audience preferences and insufficient coverage of real-world logical constraints. To overcome these issues under stringent engineering and data scarcity conditions, the authors propose a low-resource CQA approach that leverages only around one hundred expert-annotated chain-of-thought (CoT) examples. By uncovering a dual mechanism between appreciation and generation tasks within conditional generative architectures, the method enables automatic transfer of calibrated knowledge from the appreciation module to the generation module. Experimental results demonstrate that the proposed framework substantially mitigates the shortcomings of current datasets and validates the practical feasibility of aligning generative models with nuanced creative quality metrics in real-world engineering settings.