Understanding Chain-of-Thought in LLMs through Information Theory

2024-11-18

arXiv.org

AI Summary

Existing chain-of-thought (CoT) evaluation methods rely on human annotations or focus solely on final answer correctness, rendering them incapable of precisely diagnosing intermediate reasoning errors and resulting in high false positive rates. This work proposes the first unsupervised, information-theoretic CoT evaluation framework, modeling each reasoning step as an information gain process and quantifying step-wise information flow via entropy and mutual information—without requiring any labeled data—to localize reasoning bottlenecks. Its core contribution is the formalization of information gain within CoT, enabling fine-grained, interpretable failure mode identification. On benchmarks including GSM-8K, the method significantly reduces false positive rates compared to existing outcome-oriented evaluators. Moreover, it provides task-level performance attribution, uncovering latent reasoning deficiencies in models. By grounding CoT assessment in information theory, this framework establishes a novel paradigm for analyzing and improving CoT reliability.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) have shown impressive performance in complex reasoning tasks through Chain-of-Thought (CoT) reasoning, allowing models to break down problems into manageable sub-tasks. However, existing CoT evaluation techniques either require annotated CoT data or fall short in accurately assessing intermediate reasoning steps, leading to high rates of false positives. In this paper, we formalize CoT reasoning in LLMs through an information-theoretic lens. Specifically, our framework quantifies the `information gain' at each reasoning step, enabling the identification of failure modes in LLMs without the need for expensive annotated datasets. We demonstrate the efficacy of our approach through extensive experiments on toy and GSM-8K data, where it significantly outperforms existing outcome-based methods by providing more accurate insights into model performance on individual tasks.

Problem

• Assessing Chain-of-Thought reasoning without annotated data
• Quantifying information-gain in LLM reasoning steps
• Improving accuracy in evaluating intermediate reasoning performance

Innovation

• Quantifies information-gain per reasoning step
• Identifies failure modes without annotated data
• Outperforms outcome-based methods on subtasks