Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models

2026-07-30

AI Summary

This study investigates whether entropy-based pruning can effectively compress chain-of-thought (CoT) reasoning without compromising accuracy. Through cross-model and cross-task comparative experiments, token-level entropy analysis, and activation patching techniques, the authors systematically evaluate the impact of pruning low- versus high-entropy reasoning steps. Their findings indicate that entropy is not a reliable universal pruning criterion: in most tasks, entropy-guided pruning performs no better than random pruning. Only in mathematical reasoning does retaining low-entropy tokens prove beneficial—a phenomenon attributed to the intrinsically low entropy of numerical tokens rather than their reasoning value. Further patching experiments reveal that critical task information is distributed throughout the entire reasoning chain, with performance nearly fully recoverable from just a few strategically selected tokens.

Abstract

Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.

Problem

• Chain-of-Thought
• entropy-based selection
• reasoning compression
• token pruning
• large reasoning models

Innovation

• Chain-of-Thought compression
• entropy-based pruning
• reasoning distribution
• activation patching
• large reasoning models