🤖 AI Summary
The exponential growth of AI publications has rendered manual assessment of conceptual novelty inefficient and unreliable. To address this, we propose the first dual-task quantification framework for scientific evaluation—simultaneously supporting absolute classification (novel vs. non-novel) and relative ranking (cross-paper novelty comparison). Our method integrates a fine-tuned Qwen3-4B-Instruct with SciBERT, enhanced by semantic similarity modeling and contrastive learning, yielding a lightweight, interpretable, and production-ready model. Evaluated on a multi-source AI paper dataset, it significantly outperforms strong baselines including GPT-5.1. Crucially, joint optimization of both tasks improves discriminative robustness. The model is publicly released to support data-driven, scalable novelty assessment for conference reviewers and researchers alike.
📝 Abstract
With the growing ease of academic publishing, the volume of research papers, especially in AI-related fields, has surged dramatically. This flood of publications makes it difficult for truly novel and impactful work to stand out, and manual novelty assessment is often unstable and time-consuming. Our project aims to develop a model that estimates and ranks the conceptual novelty of AI papers, enabling a data-driven and scalable assessment of research originality. Such a system can help researchers efficiently identify submissions that introduce genuinely innovative ideas rather than minor variants, and provide conference reviewers with a quantitative and consistent signal of novelty. Our approach evaluates novelty primarily through a paper's title, abstract, and semantic similarity to prior literature. Given the motivation of novelty estimation, we explore two task formulations with different modeling objectives, each offering a different perspective: (1) binary classification, which predicts the paper's absolute novelty from learned patterns of prior novel works, and (2) pairwise novelty comparison, which learns to distinguish papers by relative novelty over others. We fine-tune Qwen3-4B-Instruct-2507 and SciBERT on both tasks, benchmarking against GPT-5.1 to analyze how task formulation and modeling choices affect performance. The implementation is publicly available at https://github.com/ZhengxuYan/NoveltyRank.