Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference

2026-08-04

AI Summary

This study addresses the challenges large language models face in active multi-turn abductive reasoning—specifically in evidence gathering, hypothesis updating, and termination judgment. The authors introduce an innovative “alien abduction” interactive game to systematically evaluate models’ abductive behavior under self-directed querying for the first time. Through multi-turn dialogue probes, controlled comparisons, hypothesis consistency analysis, and termination strategy assessment, the research reveals that providing evidence upfront significantly improves reasoning success rates. While models exhibit greater hypothesis consistency when selecting their own queries, they achieve lower accuracy and frequently terminate too early or issue redundant queries. These findings highlight critical limitations in the models’ capacity for active information acquisition during abductive reasoning.

Abstract

Abductive reasoning requires forming hypotheses that explain observed evidence and revising them as new evidence becomes available. While large language models (LLMs) are often evaluated on whether they solve abductive reasoning tasks correctly, less is known about how they acquire evidence, update their hypotheses, and decide when to stop. We introduce Alien Abduction game, an interactive probe for studying these behaviours under different interaction modes. The modes vary in whether evidence is provided upfront or across turns, and whether queries are selected by the model or examples are provided by the oracle. Across models, providing evidence upfront leads to higher success rates than distributing it across turns. In multi-turn settings, some models commit before using the available evidence, while others exhaust the turn budget without converging. Models also achieve higher success rates when examples are provided by the oracle than when they select their own queries, although their final hypotheses are more consistent with the evidence they selected. These findings suggest that models may form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives, and may struggle to validate and refine their hypotheses or determine when to stop.

Problem

• abductive reasoning
• active information acquisition
• multi-turn interaction
• large language models
• hypothesis revision

Innovation

• abductive reasoning
• active information acquisition
• multi-turn interaction
• large language models
• interactive probing