AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text

2024-10-05

arXiv.org

AI Summary

This study quantifies the fundamental gap in linguistic creativity between large language models (LLMs) and human professional writers, and investigates the origins and boundaries of AI creativity. Method: We propose the first computable Creativity Index, grounded in corpus attribution, which measures textual reconstructibility via web-scale corpus backtracking; we further design the DJ-SEARCH dynamic programming algorithm for efficient approximate substring matching. Contribution/Results: Empirical analysis shows human authors’ Creativity Index exceeds that of LLMs by 66.2% on average; alignment training reduces LLMs’ index by 30.1%, confirming their creativity stems primarily from corpus recitation rather than genuine originality. Our zero-shot detection method outperforms DetectGPT by 30.2% and surpasses the supervised model GhostBuster in 5 out of 6 domains, establishing a novel, attribution-based paradigm for quantifying linguistic creativity.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Creativity has long been considered one of the most difficult aspect of human intelligence for AI to mimic. However, the rise of Large Language Models (LLMs), like ChatGPT, has raised questions about whether AI can match or even surpass human creativity. We present CREATIVITY INDEX as the first step to quantify the linguistic creativity of a text by reconstructing it from existing text snippets on the web. CREATIVITY INDEX is motivated by the hypothesis that the seemingly remarkable creativity of LLMs may be attributable in large part to the creativity of human-written texts on the web. To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm that can search verbatim and near-verbatim matches of text snippets from a given document against the web. Experiments reveal that the CREATIVITY INDEX of professional human authors is on average 66.2% higher than that of LLMs, and that alignment reduces the CREATIVITY INDEX of LLMs by an average of 30.1%. In addition, we find that distinguished authors like Hemingway exhibit measurably higher CREATIVITY INDEX compared to other human writers. Finally, we demonstrate that CREATIVITY INDEX can be used as a surprisingly effective criterion for zero-shot machine text detection, surpassing the strongest existing zero-shot system, DetectGPT, by a significant margin of 30.2%, and even outperforming the strongest supervised system, GhostBuster, in five out of six domains.

Problem

• Artificial Intelligence
• Creativity Assessment
• Human-AI Comparison

Innovation

• Creativity Index
• DJ SEARCH Algorithm
• Quantitative Evaluation