The Dichotomy Between Pattern Recognition and Step-by-Step Reasoning

2026-10-06

AI Summary

This study investigates the fundamental distinctions and data dependencies between pattern recognition and step-by-step reasoning in large language models. Methodologically, reasoning trajectories are modeled as directed acyclic subgraphs based on De Bruijn graph theory. By integrating sliding window attention constraints with Qwen model fine-tuning, we systematically analyze how context windows influence reasoning capabilities. Our findings demonstrate that step-by-step reasoning exhibits sample efficiency, revealing that state frequency intrinsically governs reasoning mechanisms. We further establish that a moderate state density strategy effectively balances accuracy and robustness, while real-world data inherently manifests De Bruijn structures. Experimental results confirm that models maintain over 75% accuracy even under short context windows.

Abstract

We argue that pattern recognition and step-by-step reasoning are two ends of a spectrum. A large language model (LLM) learns to reason step-by-step when data is structured such that the next token depends on a small amount of preceding context. Inference in LLMs resembles pattern recognition when the next token depends on a large amount of preceding context. If the next token depends on only the $c$ most recent tokens, reasoning traces are paths on a De Bruijn graph whose nodes are $c$-length contexts and edges are next-token transitions between contexts. The set of reasoning traces of a task forms a directed acyclic subgraph of the De Bruijn graph. An LLM that has learned all edges of this subgraph can compose them to solve longer, unseen tasks, i.e., it reasons step-by-step. We prove that the number of edges is vanishingly small compared to the number of reasoning traces. Empirically, the number of training samples a transformer needs is a power law in the number of edges, so learning to reason step-by-step is sample efficient. We can induce De Bruijn structure in any task by maintaining a ``state'' that makes future reasoning independent of the past. The frequency of states in the reasoning trace determines $c$. We show, by fine-tuning Qwen2.5-1.5B-Instruct to solve equations and answer questions about stories, that frequent states (small $c$) result in higher accuracy but greater fragility to perturbations at test time. LLMs trained with a large $c$ are only as good as models that perform pattern recognition without reasoning. A moderate density of states balances accuracy and robustness. We show that real-world data has De Bruijn structure: Qwen3-14B and Qwen3-32B retain over 75% of their accuracy on GSM8K, MATH-500 and GPQA-Diamond when attention is restricted to a sliding window less than 15% as long as the full reasoning trace.

Problem

• pattern recognition
• step-by-step reasoning
• large language models
• De Bruijn graph
• context dependency

Innovation

• De Bruijn graph
• step-by-step reasoning
• pattern recognition
• sample efficiency
• sliding window attention