"I've Seen How This Goes": Characterizing Diversity via Progressive Conditional Surprise

2026-06-01

AI Summary

This work addresses the challenge of effectively evaluating textual diversity in AI-generated and human-written outputs to detect phenomena such as mode collapse, differences in decoding strategies, and degradation in creativity. The authors propose Decan ($D_{Ca_n}$), an unsupervised, context-learning-based diversity metric that requires only a single forward pass through a base language model. Leveraging per-token log-probabilities and information-theoretic principles, Decan operates without embeddings, reference corpora, or human annotations. It is the first method to integrate information theory with in-context learning for universal, training-free diversity assessment, enabling similarity analysis across response sets of arbitrary size. Evaluated on the McDiv benchmark, Decan achieves an OCA of 0.846 and exhibits a monotonic decline across successive post-training stages of OLMo-2-7B (SFT → DPO → RLVR), effectively capturing diversity loss in creative writing.

Abstract

Measuring the diversity of creative outputs is central to evaluating post-training mode collapse, comparing decoding strategies, and quantifying creative behavior in both AI and human writing. We propose a new approach to measuring diversity using in-context learning, of which the ``Decan'' metric, $D_{Ca_n} = C \times a_n$, is the working instance we evaluate: a per-byte score read off the per-token log-probabilities of a base model $θ$ in a \emph{single forward pass} per permutation, with no embedding model, no reference corpus, and no human labels. This approach is grounded in information theory, makes use of language model in-context learning to detect a wide range of similarities between any number of inputs, and obviates the need to train a special-purpose model. The same pipeline scores AI samples and human-written response sets, with diversity treated as a property of (responses, prompt, scoring model). On Tevet and Berant's human-grounded McDiv benchmark, $D_{Ca_n}$ reaches OCA 0.846 on the McDiv prompt\_gen set where it performs best, behind the strongest neural baseline reported in Tevet and Berant (SentBERT, 0.897). On the OLMo-2-7B post-training pipeline, $D_{Ca_n}$ drops monotonically across the base $\to$ SFT $\to$ DPO $\to$ RLVR stages, detecting the type of diversity loss that creative-writing applications care about.

Problem

• diversity measurement
• mode collapse
• creative writing
• language models
• decoding strategies

Innovation

• diversity measurement
• in-context learning
• model collapse
• information theory
• Decan metric