novelty assessment

This work addresses the high cost, subjectivity, and limited scalability of human evaluation of research idea novelty, as well as the absence of a unified benchmark for automated assessment methods. To this end, we introduce RINoBench, the first large-scale benchmark for novelty judgment, comprising 1,381 research ideas annotated by human experts alongside nine automatic evaluation metrics. We systematically evaluate large language models (LLMs) on both scoring accuracy and reasoning plausibility. Our experiments reveal that while LLM-generated rationales closely resemble human reasoning, their novelty judgments significantly deviate from human gold-standard assessments, exposing a critical disconnect between reasoning processes and actual judgment capabilities in current models. RINoBench provides a standardized, scalable framework for evaluating research idea novelty.

Current approaches to assessing the novelty of scientific papers rely heavily on subjective judgment and lack systematic, interpretable, and reviewer-aligned objective methods. This work proposes the first knowledge-driven framework for novelty evaluation, explicitly modeling human judgments of novelty derived from peer review comments across nearly 80,000 top-tier AI conference papers. By integrating structured paper representations with a semantic similarity graph of related literature, the framework enables fine-grained, concept-level originality comparisons. It combines large language model fine-tuning, knowledge extraction, and semantic retrieval to produce calibrated, interpretable novelty scores that significantly outperform existing methods in accuracy, consistency, and alignment with human reviewers.

This work addresses the challenge of efficiently and objectively evaluating the novelty of scholarly submissions in peer review, particularly amidst the rapidly expanding volume of scientific literature. To this end, we propose an intelligent agent system powered by large language models that implements a four-stage pipeline—contribution extraction, semantic retrieval, hierarchical classification coupled with fine-grained full-text comparison, and evidence synthesis—to deliver an end-to-end, traceable, and evidence-based novelty assessment grounded in actual published works. This approach effectively mitigates hallucination risks inherent in large language models. Deployed on over 500 submissions to ICLR 2026, our method accurately identifies relevant prior work omitted by authors, significantly enhancing the fairness, consistency, and interpretability of peer reviews. All evaluation reports have been publicly released.

To address the time-consuming, subjective, and non-scalable nature of manual literature reviews in assessing scientific idea novelty, this paper proposes a retrieval-augmented generation (RAG) framework. Methodologically, it employs a two-stage retrieval process coupled with a facet-aware large language model (LLM) re-ranking mechanism, integrating keyword/phrase matching, embedding-based filtering, and literature-anchored generative reasoning; expert-annotated examples further enhance interpretability. The core contribution lies in incorporating structured bibliographic facets—such as methodology, problem domain, and technical approach—into re-ranking, enabling fine-grained, traceable novelty assessment. Experiments demonstrate a ~13% improvement in inter-annotator agreement for novelty classification over baseline methods. Ablation studies confirm the critical role of the facet-aware re-ranking module, significantly boosting identification of highly relevant prior work and improving system robustness.

This study addresses the narrow scope of purely theory- or data-driven approaches in AI-assisted innovation by proposing the first LLM-based hypothesis generation framework that jointly leverages scholarly literature and empirical data. Methodologically, it introduces a novel dual-source synergy mechanism integrating literature semantic parsing with multi-source data alignment, augmented by domain-specific prompt engineering and rigorous human evaluation. Key contributions include: (1) establishing the first theory- and data-coordinated paradigm for automated hypothesis generation; (2) achieving statistically significant improvements in hypothesis quality—+8.97% over few-shot baselines, +15.75% over literature-only methods, and +3.37% over data-only methods—across five benchmark datasets; and (3) demonstrating via human evaluation that the framework substantially enhances decision-making accuracy in AI-content identification tasks, with gains ranging from 7.44% to 14.19%.

This work addresses the lack of verifiable and traceable mechanisms for assessing biological novelty in current large language model–based scientific agents, which undermines scientific validity. The authors propose Plato-Bio, a system that integrates explicit workflow states, claim–evidence linking, citation verification, scoped file writing, and publication gating into the Plato/Denario architecture. For the first time, it embeds software contracts and auditable benchmarks into the biological discovery pipeline, mitigating three critical evaluation flaws: task-domain drift, missing methodological signals, and incomplete evidence denominators. Built on a Python workflow engine, Plato-Bio incorporates TF-IDF ranking, AlphaFold structural alignment, RMSD computation, and confidence masking. Evaluations demonstrate accurate retrospective rediscovery of the fish oil–Raynaud’s phenomenon link, Cα-RMSD values below 1 Å for 11 of 15 proteins, 27 traceable unvalidated hypothesis regions, and 100% test suite pass rates.

This study investigates the reliability of large language models (LLMs) as judges in assessing the novelty of scientific research questions. Addressing the lack of objective benchmarks in existing evaluations, the authors propose anchoring assessments to the original research questions formulated by human authors and introduce RQ-Bench, a novel dataset for systematic evaluation. Through comprehensive experiments combining independent scoring, pairwise comparisons, and expert human judgments, the work reveals for the first time a pervasive “novelty hallucination” in LLMs—whereby they consistently overestimate the novelty of model-generated questions compared to those posed by human researchers, whom domain experts markedly favor. This research not only exposes a critical limitation of current LLMs in scientific reasoning but also establishes a new paradigm for evaluating novelty at the level of research questions.

The exponential growth of AI publications has rendered manual assessment of conceptual novelty inefficient and unreliable. To address this, we propose the first dual-task quantification framework for scientific evaluation—simultaneously supporting absolute classification (novel vs. non-novel) and relative ranking (cross-paper novelty comparison). Our method integrates a fine-tuned Qwen3-4B-Instruct with SciBERT, enhanced by semantic similarity modeling and contrastive learning, yielding a lightweight, interpretable, and production-ready model. Evaluated on a multi-source AI paper dataset, it significantly outperforms strong baselines including GPT-5.1. Crucially, joint optimization of both tasks improves discriminative robustness. The model is publicly released to support data-driven, scalable novelty assessment for conference reviewers and researchers alike.

This study addresses the limitations of existing scientific novelty metrics, which often conflate novelty with impact or quality by relying on proxies such as citations or peer review, and lack reliable evaluation benchmarks. To overcome this, the authors propose an axiomatic framework grounded in scientific norms for measuring novelty and introduce a benchmark comprising ten tasks spanning three AI subfields, enabling the first systematic, axiom-based evaluation of novelty metrics. Experimental results demonstrate that no existing metric consistently satisfies all proposed axioms; however, by integrating complementary metrics through a fusion architecture, benchmark performance improves significantly from 71.5% to 90.1%, thereby validating the effectiveness and superiority of the proposed approach.

Detecting novelties in high-dimensional, noisy scientific big data remains challenging, and existing anomaly detection methods often lack statistical rigor in declaring anomalies. Method: This paper proposes an end-to-end framework integrating contrastive representation learning with nonparametric two-sample hypothesis testing. It innovatively combines contrastive pre-trained embeddings with the New Physics Learning Machine (NPLM), augmented by simulation-based data augmentation and low-dimensional feature extraction. Contribution/Results: The framework enables highly sensitive detection of subtle anomalous signals and supports quantifiable, p-value-driven scientific discovery. Evaluated on real and synthetic datasets across astronomy, physics, and biology, it significantly outperforms state-of-the-art anomaly detection methods, demonstrating strong robustness to noise, interpretability via statistically grounded inference, and cross-domain generalizability.