Thinking Under Uncertainty: Evidence Use and Information-Seeking in Language Models

2026-07-29

AI Summary

This study investigates whether large language models, during reasoning, more effectively utilize existing evidence or actively seek new information to optimize decisions. Using a two-armed bandit task under uncertainty, the authors compare model behavior in “thinking” versus “non-thinking” modes, integrating cognitive modeling with decoder parameter sweeps to disentangle metacognitive monitoring and control signals in a controlled setting for the first time. Results show that thinking primarily enhances value-guided action selection, reduces choice noise unrelated to uncertainty, and increases confidence sensitivity to task difficulty and evidence strength, but does not significantly promote information seeking. The analysis further identifies both UCB-like and Thompson-like exploration strategies, revealing that thinking optimizes the exploitation of current evidence rather than driving active exploration.

Abstract

Inference-time thinking improves the performance of large language models, but aggregate outcomes do not reveal whether models use available evidence more effectively or seek information that could improve future decisions. We distinguish these responses by measuring action preference, thinking length, and reported confidence under matched uncertainty. Ten open-weight models completed matched horizon-style two-armed bandit trials in thinking and non-thinking modes. A cognitive model separated value-guided action and uncertainty-independent choice noise from two behavioral signatures of exploration: a UCB-like preference for the less-known arm and Thompson-like choice variability that increases with total uncertainty. On average, thinking strengthened value-guided action and reduced uncertainty-independent choice noise, without producing UCB-like exploration or strengthening Thompson-like exploration. Outside action, the information-imbalanced history condition, which also displayed more observations than the matched balanced condition, was associated with greater thinking length. Reported confidence became more sensitive to decision difficulty and more strongly associated with chosen task evidence. We interpret these thinking-length and reported-confidence patterns as consistent with metacognitive control and metacognitive monitoring, respectively, without establishing either process. Decoder sweeps, especially temperature, altered choice noise and thinking length but did not reproduce the joint cross-output pattern. In this controlled decision setting, thinking improved how models acted on current evidence, while neither measured signature supported a shift toward a more information-seeking policy.

Problem

• language models
• uncertainty
• information-seeking
• evidence use
• decision-making

Innovation

• inference-time thinking
• information-seeking
• metacognitive monitoring
• two-armed bandit
• cognitive modeling