Chain-of-Probe: Examing the Necessity and Accuracy of CoT Step-by-Step

2024-06-23

arXiv.org

AI Summary

Large language models (LLMs) frequently exhibit “early answering”—producing final answers before generating Chain-of-Thought (CoT) reasoning—raising fundamental questions: Is CoT necessary? Does answer correctness imply correct reasoning? Method: The authors introduce Chain-of-Probe, the first framework to quantitatively decouple CoT necessity from reasoning accuracy. It employs dynamic neuron probing, inter-layer state difference analysis, and step-wise reasoning modeling to assess when and why CoT is required. Contributions/Results: Chain-of-Probe reveals that over 50% of correct answers stem from flawed reasoning; establishes a strong correlation between task complexity and CoT necessity; and enables answer re-ranking based on reasoning trustworthiness. Evaluated on GSM8K and other benchmarks, it improves reasoning trustworthiness by 23%, significantly enhancing both interpretability and reliability of LLM outputs.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Current research found the issue of Early Answering in large language models (LLMs), where the models already have an answer before generating the Chain-of-Thought (CoT). This phenomenon suggests a potential lack of necessary dependency between the predicted answer and the reasoning process. Consequently, two important questions arise: (1) Is CoT still necessary if the model already has an answer? (2) Can the correctness of the answer serve as valid evidence for the correctness of CoT? To address these questions, we propose a method, namely Chain-of-Probe (CoP), to probe changes in the mind during the model's reasoning. The probing results show that in a significant number of question-answer cases, CoT appears to be unnecessary, and this necessity correlates with the simplicity of the task, defined by reasoning steps required. Furthermore, by analyzing patterns in mind change, we examine the correctness of the model's reasoning. Our validation reveals that many responses, although correct in their final answer, contain errors in their reasoning process. To this end, we propose a strategic approach based on CoP to prioritize answers with correct reasoning among multiple candidates, thereby bolstering the reliability of the model's reasoning.

Problem

• Examine necessity of Chain-of-Thought in LLMs
• Assess correctness of reasoning via Chain-of-Probe
• Prioritize answers with correct reasoning processes

Innovation

• Chain-of-Probe examines reasoning necessity.
• Probes mind changes during model reasoning.
• Prioritizes answers with correct reasoning steps.