When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

2026-07-21

AI Summary

This work investigates the collapse of action diversity in large language models during supervised fine-tuning (SFT) for sequential decision-making, which undermines exploration capabilities. Through state-level evaluation, adversarial play, and analysis of training trajectories in a deterministic Tic-Tac-Toe variant, the study identifies narrow-support imitation as the root cause of policy collapse and reveals that reasoning processes often suppress diversity without necessarily improving accuracy. To address this, the authors propose an action-augmented SFT approach that preserves the support of the action distribution while maintaining exploratory behavior. Experiments demonstrate that while standard SFT improves accuracy at the cost of significant diversity loss, the proposed method effectively mitigates this degradation while retaining competitive performance.

Abstract

Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks, but its effect on behavioral diversity in sequential decision-making remains under-explored. We study this question in a controlled suite of deterministic board games based on tic-tac-toe variants, where optimal actions are exactly computable and diversity can be measured directly. Across state-level evaluation, arena gameplay, and training trajectories, we find that reasoning-mode generation frequently suppresses action diversity without uniformly improving action accuracy. Furthermore, standard SFT improves accuracy but often induces premature diversity collapse, which exceeds what is minimally required by the accuracy-diversity tradeoff. We then show that action augmentation, which trains on all optimal actions per state rather than a single demonstrated action, would partially mitigates this effect. Our results identify narrow-support imitation as a source of policy collapse in LLM decision-making and suggest that preserving action support during SFT is important for maintaining exploratory behavior.

Problem

• behavioral diversity
• supervised fine-tuning
• sequential decision-making
• policy collapse
• action diversity

Innovation

• diversity collapse
• supervised fine-tuning
• action augmentation
• narrow-support imitation
• behavioral diversity