Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning

2026-06-05

AI Summary

This work addresses the “concept bottleneck” in existing Continuous Conceptual Thought (CoCoNuT) methods, where intermediate latent states are overwritten during multi-step reasoning, leading to performance degradation as reasoning depth increases. To overcome this limitation, the authors propose AGCLR, the first approach to integrate a persistent memory mechanism into continuous latent reasoning frameworks. AGCLR employs a three-gated architecture—comprising write, read, and forget gates—to dynamically manage residual memory across reasoning steps, thereby preserving and leveraging critical intermediate facts over multiple inference rounds. Implemented on a GPT-2 backbone, AGCLR consistently outperforms baseline models on GSM8K, HotpotQA, and ProsQA benchmarks. Notably, its performance gains amplify with increasing reasoning depth, effectively mitigating the concept bottleneck.

Abstract

Large language models (LLMs) have demonstrated remarkable reasoning abilities on mathematical and multi-hop planning tasks. The CoCoNuT (Chain of Continuous Thought) paradigm~\cite{hao2024coconut} extends this by enabling models to reason in latent space, exploring multiple reasoning paths simultaneously rather than committing to a single chain early on. However, we identify a limitation we term the \textbf{concept bottleneck}. At each reasoning pass, intermediate hidden states are overwritten, causing the model to lose critical facts computed in earlier steps as reasoning depth increases. We observe this empirically. On HotpotQA, vanilla CoCoNuT (10.4\% EM) fails to improve over the CoT baseline (11.0\% EM), and performance degrades with curriculum depth on GSM8K. To address this, we propose \textbf{AGCLR} (Adaptive Gated Continuous Latent Reasoning), which augments CoCoNuT with a \textit{Gated Concept Stream}. A persistent residual memory maintained across all reasoning passes, controlled by three learned gates: a \textit{write} gate that commits intermediate facts to memory, a \textit{read} gate that retrieves relevant prior states, and a \textit{forget} gate that prunes irrelevant context. Evaluated on GSM8K, HotpotQA, and ProsQA using GPT-2 as our base model, AGCLR achieves consistent improvements across all types of datasets. With the performance gap compounding as curriculum depth increases, directly resolving the concept bottleneck. Code available at https://anonymous.4open.science/r/JJJJ/README.md

Problem

• concept bottleneck
• continuous latent reasoning
• residual memory
• multi-hop reasoning
• intermediate state retention

Innovation

• AGCLR
• Gated Concept Stream
• Persistent Memory
• Continuous Latent Reasoning
• Concept Bottleneck