DeFine: Enhancing LLM Decision-Making with Factor Profiles and Analogical Reasoning

2024-10-02

arXiv.org

AI Summary

Large language models (LLMs) suffer from decision bias when processing speech-to-text transcripts—such as earnings calls, clinical consultations, and debates—due to pervasive grammatical incompleteness, semantic ambiguity, and subjective uncertainty. Method: We propose a scenario-driven, probabilistic factor profiling and analogical reasoning decoupling architecture: (1) an interpretable, uncertainty-aware factor profiler for probabilistic modeling of ambiguous expressions; (2) cross-context analogical retrieval to enhance reasoning consistency; and (3) prompt engineering integrated with uncertainty-calibrated fine-tuning. Contribution/Results: Our approach achieves the first decoupled yet synergistic integration of uncertainty quantification and decision modeling. Evaluated on earnings call analysis, clinical consultation, and debate analysis, it improves decision accuracy by 12.7% and reduces uncertainty calibration error by 34.5%, significantly enhancing robustness and interpretability in high-uncertainty scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLMs are ideal for decision-making due to their ability to reason over long contexts and identify critical factors. However, challenges arise when processing transcripts of spoken speech describing complex scenarios. These transcripts often contain ungrammatical or incomplete sentences, repetitions, hedging, and vagueness. For example, during a company's earnings call, an executive might project a positive revenue outlook to reassure investors, despite significant uncertainty regarding future earnings. It is crucial for LLMs to incorporate this uncertainty systematically when making decisions. In this paper, we introduce DeFine, a new framework that constructs probabilistic factor profiles from complex scenarios. DeFine then integrates these profiles with analogical reasoning, leveraging insights from similar past experiences to guide LLMs in making critical decisions in novel situations. Our framework separates the tasks of quantifying uncertainty in complex scenarios and incorporating it into LLM decision-making. This approach is particularly useful in fields such as medical consultations, negotiations, and political debates, where making decisions under uncertainty is vital.

Problem

• Handling verbose speech transcripts with repetition and vagueness
• Systematically incorporating uncertainty in LLM decision-making
• Making critical decisions in new situations using analogical reasoning

Innovation

• Modular framework constructs probabilistic factor profiles
• Integrates profiles with analogical reasoning
• Separates uncertainty quantification from decision-making