Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

2026-10-01

AI Summary

This study addresses the pronounced instability of large language models (LLMs) when employed as novelty evaluators, wherein minor prompt variations can induce substantial shifts in judgment. To investigate this, we construct an extreme-consensus dataset derived from OpenReview and conduct pairwise comparison experiments alongside controlled multi-baseline testing to systematically examine how critical factors such as prompt design influence LLM-based novelty assessment. Our findings reveal for the first time that slight prompt modifications can cause evaluation accuracy to fluctuate by over 50 percentage points, and that specialized evaluators consistently underperform simple baselines. These results challenge the validity of existing automated evaluation frameworks for creative systems and underscore the urgent need for more robust evaluation paradigms.

Abstract

Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We first build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.

Problem

• novelty evaluation
• large language models
• automated ideation
• evaluation instability
• prompt sensitivity

Innovation

• Novelty Evaluation
• Large Language Models
• Automated Ideation
• Prompt Sensitivity
• Evaluation Instability