Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability

2025-10-07

AI Summary

Instruction tuning of language models improves instruction-following capability but often degrades distribution modeling quality in multi-answer tasks—specifically, context controllability, effective output space coverage, and distribution alignment. This work identifies a key flaw in current approaches: conflating the elicitation of pretrained priors with genuine context-guided generation. To address this, we propose Spectrum Tuning, a novel post-training paradigm that jointly optimizes these three desiderata. We introduce the Spectrum Suite, a comprehensive evaluation benchmark incorporating diverse target distributions—including human preferences and numerical distributions—to rigorously assess distributional fidelity. By explicitly modeling conditional distributions, our method enhances dynamic context responsiveness and output diversity. Experiments demonstrate that Spectrum Tuning significantly improves distribution alignment, output coverage breadth, and context-controllability accuracy on unseen datasets—while preserving pretraining performance across standard benchmarks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Language model post-training has enhanced instruction-following and performance on many downstream tasks, but also comes with an often-overlooked cost on tasks with many possible valid answers. We characterize three desiderata for conditional distributional modeling: in-context steerability, valid output space coverage, and distributional alignment, and document across three model families how current post-training can reduce these properties. In particular, we disambiguate between two kinds of in-context learning: ICL for eliciting existing underlying knowledge or capabilities, and in-context steerability, where a model must use in-context information to override its priors and steer to a novel data generating distribution. To better evaluate and improve these desiderata, we introduce Spectrum Suite, a large-scale resource compiled from >40 data sources and spanning >90 tasks requiring models to steer to and match diverse distributions ranging from varied human preferences to numerical distributions and more. We find that while current post-training techniques help elicit underlying capabilities and knowledge, they hurt models' ability to flexibly steer in-context. To mitigate these issues, we propose Spectrum Tuning, a post-training method using Spectrum Suite to improve steerability and distributional coverage. We find that Spectrum Tuning often improves over pretrained models and their instruction-tuned counterparts, enhancing steerability, spanning more of the output space, and improving distributional alignment on held-out datasets.

Problem

• Evaluating post-training impact on model distributional coverage and steerability
• Addressing reduced flexibility in adapting to novel data distributions
• Improving alignment with diverse human preferences and numerical distributions

Innovation

• Spectrum Tuning method enhances steerability and coverage
• Post-training technique improves distributional alignment on datasets
• Uses Spectrum Suite with diverse tasks for training