Ehrenfeucht-Haussler Rank and Chain of Thought

2025-01-22

AI Summary

What structural complexity measure characterizes the minimal chain-of-thought (CoT) steps required to compute a Boolean function? Method: The authors introduce the Ehrenfeucht–Haussler (EH) rank—a classical concept from computational learning theory—into the theoretical analysis of large language models, and rigorously analyze single-layer hard-attention Transformer decoders. Contribution/Results: They prove that the minimum number of CoT steps needed for such a decoder to compute any Boolean function equals its EH rank—establishing an exact equivalence. This yields tight lower bounds on CoT length and precisely determines that ℓ-repeated composition and k-th order 1-localization tasks require exactly ℓ and k CoT steps, respectively. The work provides the first theoretically optimal characterization of CoT step complexity for structured reasoning tasks, demonstrating that CoT is inherently incompressible and non-redundant with respect to intrinsic Boolean function complexity.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The notion of rank of a Boolean function has been a cornerstone in the theory of PAC learning, enabling quasipolynomial-time learning algorithms for polynomial-size decision trees. We present a novel characterization of rank, grounded in the well-known Transformer architecture. We show that the rank of a function $f$ corresponds to the minimum number of Chain of Thought (CoT) steps required by a single-layer transformer decoder with hard attention to compute $f$. Based on this characterization we establish tight bounds on the number of CoT steps required for specific problems, showing that $ell$-fold function composition necessitates exactly $ell$ CoT steps. Furthermore, we analyze the problem of identifying the position of the $k$-th occurrence of 1 in a Boolean sequence, proving that it requires $k$ CoT steps.

Problem

• Boolean function complexity
• Transformer architecture
• Decision tree learning efficiency

Innovation

• Boolean Function Rank
• Transformer Architecture
• Chain of Thought (CoT)