Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

2026-06-10

AI Summary

Existing approaches to creativity evaluation are often confined to specific tasks, lacking generality and scalability. This work proposes the first task-agnostic automated evaluation framework that decouples creativity measurement from any particular task: it quantifies divergent creativity—encompassing novelty and diversity—using semantic entropy, and introduces a retrieval-augmented multi-agent adjudication mechanism to assess convergent creativity. Validated across three heterogeneous benchmarks (MacGyver, HypoGen, and BookMIA), the framework reliably captures core dimensions of creativity while improving evaluation efficiency by over 60%. Furthermore, it offers new insights into how model scale, temperature, and reasoning capabilities influence creative performance.

Abstract

Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.

Problem

• creativity evaluation
• large language models
• open-ended tasks
• domain-agnostic
• automated assessment

Innovation

• domain-agnostic evaluation
• semantic entropy
• multi-agent judge
• divergent creativity
• convergent creativity