One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies

2026-07-23

AI Summary

This work addresses the limitation of existing large language models in multi-turn clarification under ambiguous user requests, which typically prioritize final accuracy while neglecting interaction efficiency and decision quality. The authors propose RegretBench, a novel benchmark that formulates clarification as a sequential decision-making problem. By incorporating hidden user intents, semantic state tracking, and a regret-based objective function, RegretBench jointly evaluates a model’s strategic competence in determining when to ask questions, what content to query, and when to terminate clarification. Built upon free-form multi-turn dialogues and assessed relative to reference policies, the framework reveals significant disparities among models in efficiency, robustness, and stopping decisions—even when their final success rates are comparable—highlighting that effective clarification hinges on precise timing of inquiries and timely termination.

Abstract

Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a multi-turn benchmark that evaluates clarification as policy behavior rather than isolated question quality. RegretBench provides a hidden-intent formulation of ambiguity, supports free-form interaction grounded in semantic-state tracking, and introduces a regret-based objective that measures how much value a model loses relative to a reference clarification policy. Experiments on open-domain QA and product recommendation scenarios show that final success alone is insufficient, as models with similar accuracy can differ substantially in efficiency, robustness to user behaviors, and stopping decisions. By jointly measuring intent resolution, interaction cost, ineffective clarification, and regret, RegretBench reveals whether models clarify usefully and efficiently. Our results show that effective clarification requires more than plausible questions: models must ask the right question at the right time and stop once the user's intended meaning is clear.

Problem

• clarification
• multi-turn dialogue
• regret
• ambiguous requests
• conversational LLMs

Innovation

• regret-based evaluation
• clarification policy
• multi-turn benchmark
• semantic-state tracking
• hidden-intent ambiguity