Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate

2025-07-16

AI Summary

To address semantic ambiguity in user queries that leads to misinterpretation by large language models (LLMs), this paper proposes a multi-agent debate framework for ambiguity identification and resolution. The framework orchestrates heterogeneous LLMs—Llama3-8B, Gemma2-9B, and Mistral-7B—through a structured debate protocol, explicitly modeling disagreements, exchanging evidential reasoning traces, and dynamically converging toward consensus. Unlike single-model fine-tuning or prompt engineering, our approach reframes ambiguity handling as a collaborative, verifiable multi-agent reasoning process, substantially enhancing robustness in complex semantic parsing tasks. Experimental results show that the Mistral-7B–driven debate system achieves a 76.7% consensus success rate on ambiguity identification, significantly outperforming single-model baselines—particularly on challenging cases involving lexical polysemy, implicit intent, and context-dependent requests.

Technology Category

• Multiagent Systems: Agreement, Argumentation & Negotiation
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have demonstrated significant capabilities in understanding and generating human language, contributing to more natural interactions with complex systems. However, they face challenges such as ambiguity in user requests processed by LLMs. To address these challenges, this paper introduces and evaluates a multi-agent debate framework designed to enhance detection and resolution capabilities beyond single models. The framework consists of three LLM architectures (Llama3-8B, Gemma2-9B, and Mistral-7B variants) and a dataset with diverse ambiguities. The debate framework markedly enhanced the performance of Llama3-8B and Mistral-7B variants over their individual baselines, with Mistral-7B-led debates achieving a notable 76.7% success rate and proving particularly effective for complex ambiguities and efficient consensus. While acknowledging varying model responses to collaborative strategies, these findings underscore the debate framework's value as a targeted method for augmenting LLM capabilities. This work offers important insights for developing more robust and adaptive language understanding systems by showing how structured debates can lead to improved clarity in interactive systems.

Problem

• Enhancing LLM detection of ambiguity in user requests
• Improving resolution capabilities beyond single models
• Structured debates for clearer interactive systems

Innovation

• Multi-agent debate framework enhances ambiguity detection
• Three LLM architectures collaborate for better resolution
• Structured debates improve clarity in interactive systems