CA-BED: Conversation-Aware Bayesian Experimental Design

2026-05-31

AI Summary

This work addresses the challenge that large language models (LLMs) struggle to efficiently reason under ambiguity or partial information in interactive question-answering scenarios. To overcome this limitation, the authors propose a probabilistic dialogue planning framework at inference time that integrates Bayesian experimental design with LLM-based likelihood estimation. By maintaining a belief distribution over possible states, simulating dialogue trees, and propagating expected information gain, the method proactively selects optimal multi-turn questions to maximize information acquisition. Evaluated on two entity-reasoning benchmarks, the approach substantially outperforms standard prompting strategies, achieving an average success rate improvement of 21.8% with only 1.8 additional dialogue turns, thereby demonstrating both high efficiency and practical applicability.

Abstract

Large Language Models (LLMs) excel at static reasoning tasks, yet their performance often degrades in interactive scenarios where information must be actively acquired through questioning. A key challenge lies in selecting questions that reduce uncertainty while incorporating responses that may be ambiguous or only partially informative. To address this, we propose Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time probabilistic dialog planning framework that integrates Bayesian Experimental Design with LLM-based likelihood estimation to optimize question selection over multiple conversational turns. CA-BED maintains a belief distribution over hypotheses, anticipates possible answers, and propagates expected information gain through a simulated conversation tree. Across two structured entity-deduction benchmarks, CA-BED yields an average 21.8% improvement in success rates over direct prompting, with comparable gains relative to alternative information-seeking methods. It achieves these gains with an average increase of only 1.8 conversational turns compared to direct prompting.

Problem

• interactive questioning
• uncertainty reduction
• ambiguous responses
• information acquisition
• conversational reasoning

Innovation

• Bayesian Experimental Design
• Large Language Models
• Active Questioning
• Conversational Reasoning
• Information Gain