Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models

2025-07-28

AI Summary

Instruction tuning substantially reduces output diversity in large language models (LLMs), hindering creative text generation. This work systematically quantifies the “diversity gap” across multiple open-source LLMs on narrative generation tasks and identifies the DPO fine-tuning stage as the primary source of diversity degradation. To address this, we propose **Conformal Decoding**, an inference-time technique that leverages the base model’s token probability distribution to guide sampling in instruction-tuned models, thereby restoring lexical and syntactic diversity. Experiments on OLMo and OLMo 2 demonstrate that our method increases n-gram entropy by +23.6%—a strong indicator of enhanced diversity—while preserving or even improving generation quality (BLEU +1.4, human evaluation +0.8/5). Crucially, Conformal Decoding requires no retraining, is computationally lightweight, and scales seamlessly across models, offering a practical, deployable solution for high-creativity generation tasks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Instruction-tuning large language models (LLMs) reduces the diversity of their outputs, which has implications for many tasks, particularly for creative tasks. This paper investigates the ``diversity gap'' for a writing prompt narrative generation task. This gap emerges as measured by current diversity metrics for various open-weight and open-source LLMs. The results show significant decreases in diversity due to instruction-tuning. We explore the diversity loss at each fine-tuning stage for the OLMo and OLMo 2 models to further understand how output diversity is affected. The results indicate that DPO has the most substantial impact on diversity. Motivated by these findings, we present a new decoding strategy, conformative decoding, which guides an instruct model using its more diverse base model to reintroduce output diversity. We show that conformative decoding typically increases diversity and even maintains or improves quality.

Problem

• Instruction-tuning reduces output diversity in LLMs
• Diversity gap exists in narrative generation tasks
• Conformative decoding reintroduces diversity while maintaining quality

Innovation

• Conformative decoding enhances output diversity
• Uses base model to guide instruct model
• Maintains or improves output quality