InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

2026-02-16

AI Summary

Existing approaches to evaluating scientific creativity are constrained by narrow knowledge horizons, single-dimensional criteria, and inherent biases in large language models, rendering them inadequate for the rapidly growing volume of novel ideas. This work proposes InnoEval, a novel framework that formalizes creativity assessment as a knowledge-grounded, multi-perspective reasoning task. InnoEval leverages a heterogeneous deep-knowledge search engine to retrieve dynamic, multi-source evidence and employs a multi-disciplinary agent-based review panel to conduct decoupled, multi-dimensional evaluations, thereby simulating the collective deliberation process of human experts. Experimental results demonstrate that InnoEval significantly outperforms baseline methods across pairwise, paired, and group-wise evaluation tasks, with its judgments showing strong alignment with expert consensus, thus validating the framework’s effectiveness and reliability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

Problem

• idea evaluation
• knowledge grounding
• multi-perspective reasoning
• LLM-as-a-Judge
• scientific innovation

Innovation

• knowledge-grounded reasoning
• multi-perspective evaluation
• heterogeneous knowledge retrieval
• LLM-as-a-Judge mitigation
• innovation review board