Conformal Information Pursuit for Interactively Guiding Large Language Models

2025-07-03

AI Summary

To address suboptimal query selection, redundant interaction rounds, and degraded performance in interactive question-answering with large language models (LLMs)—stemming from miscalibrated output probabilities—this paper proposes a conformal prediction-based sequential query optimization method. The core innovation lies in dynamically estimating conditional entropy via the size of conformal prediction sets, replacing conventional information gain to establish a distribution-free, robust uncertainty quantification mechanism. This measure is integrated into an information-seeking framework to enable adaptive query policy optimization. Evaluated on the 20 Questions and MediQ benchmarks, the method significantly reduces average dialogue length while achieving prediction accuracy comparable to single-turn inference. Moreover, it enhances policy stability and interpretability without requiring model retraining or access to ground-truth labels.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

A significant use case of instruction-finetuned Large Language Models (LLMs) is to solve question-answering tasks interactively. In this setting, an LLM agent is tasked with making a prediction by sequentially querying relevant information from the user, as opposed to a single-turn conversation. This paper explores sequential querying strategies that aim to minimize the expected number of queries. One such strategy is Information Pursuit (IP), a greedy algorithm that at each iteration selects the query that maximizes information gain or equivalently minimizes uncertainty. However, obtaining accurate estimates of mutual information or conditional entropy for LLMs is very difficult in practice due to over- or under-confident LLM probabilities, which leads to suboptimal query selection and predictive performance. To better estimate the uncertainty at each iteration, we propose Conformal Information Pursuit (C-IP), an alternative approach to sequential information gain based on conformal prediction sets. More specifically, C-IP leverages a relationship between prediction sets and conditional entropy at each iteration to estimate uncertainty based on the average size of conformal prediction sets. In contrast to conditional entropy, we find that conformal prediction sets are a distribution-free and robust method of measuring uncertainty. Experiments with 20 Questions show that C-IP obtains better predictive performance and shorter query-answer chains compared to previous approaches to IP and uncertainty-based chain-of-thought methods. Furthermore, extending to an interactive medical setting between a doctor and a patient on the MediQ dataset, C-IP achieves competitive performance with direct single-turn prediction while offering greater interpretability.

Problem

• Optimize sequential querying to minimize expected queries in LLMs
• Improve uncertainty estimation for better query selection in LLMs
• Enhance predictive performance and interpretability in interactive LLM tasks

Innovation

• Uses conformal prediction sets for uncertainty estimation
• Minimizes queries via sequential information gain strategy
• Enhances interpretability in interactive LLM applications