Towards Robust Dialogue Breakdown Detection: Addressing Disruptors in Large Language Models with Self-Guided Reasoning

2025-04-26

AI Summary

This work addresses dialogue breakdowns in LLM-based conversational systems caused by incoherent or logically inconsistent responses. We propose an interpretable, robust framework for real-time detection and mitigation. Methodologically, we introduce a novel self-guided reasoning mechanism, integrating supervised fine-tuning of an 8B-parameter model, multi-strategy prompt engineering—including chain-of-thought prompting, analogical prompting, and confidence calibration—and a resource-efficient deployment architecture that dynamically escalates to larger models only when necessary. Our contributions are threefold: (1) the first high-accuracy, small-model-driven dialogue breakdown detector with strong cross-lingual generalization across English and Japanese; (2) a 7% absolute accuracy improvement on the BETOLD benchmark, substantially narrowing the performance gap between open-source small models and proprietary large models; and (3) a new paradigm for lightweight, trustworthy conversational systems grounded in interpretability and efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) are rapidly changing various domains. However, their capabilities in handling conversational breakdowns still require an in-depth exploration. This paper addresses the challenge of detecting and mitigating dialogue breakdowns within LLM-driven conversational systems. While powerful models from OpenAI and Anthropic excel in many dialogue tasks, they can still produce incoherent or contradictory responses, commonly referred to as breakdowns, which undermine user trust. To tackle this, we propose an approach that combines specialized fine-tuning with advanced prompting strategies, including few-shot learning, chain-of-thought reasoning, and analogical prompting. In particular, we fine-tune a small 8B model and demonstrate its robust classification and calibration capabilities in English and Japanese dialogue. We also validate its generalization on the BETOLD dataset, achieving a 7% accuracy improvement over its base model. Furthermore, we introduce a real-time deployment architecture that selectively escalates suspicious responses to more resource-intensive frontier models only when breakdowns are detected, significantly cutting operational expenses and energy consumption. Experimental results show our method surpasses prior state-of-the-art specialized classifiers while also narrowing performance gaps between smaller open-source models and large proprietary ones. Our approach offers a scalable solution for robust conversational AI in high-impact domains by combining efficiency, interpretability, and reliability.

Problem

• Detecting and mitigating dialogue breakdowns in LLM-driven systems
• Improving model robustness via fine-tuning and advanced prompting strategies
• Reducing operational costs with selective real-time breakdown escalation

Innovation

• Combines fine-tuning with advanced prompting strategies
• Introduces real-time deployment architecture for efficiency
• Improves accuracy and calibration in multilingual dialogues